James Brundage | MVP

I constantly feel lucky to have started learned computing when I did (late 80s-present).

Things were simple enough that you learned the foundation.

And could truly enjoy things getting easier with time.

Understanding runs deep.

There's no shallow end to the pool anymore.

I feel like not wanting to be stuck in a sandbox is the start of the long slow road to being a power user.

Of course, nowadays, you can run Linux on Android and install PowerShell on a phone. (but good luck typing in that window)

I feel like the original post explains why this is:

a) lots less boilerplate and repetition in PowerShell
b) far fewer PowerShell modules than websites in the world
c) PowerShell code has varying functions of varying scope

Will add:

d) PowerShell code quality varies _wildly_

Yes, I think that sums it up nicely.

The ethical problem I have / see is that each time this happens, vast quantities of information disappear into the ether.

The "big thing" at proto / xrpc are designed to fix is this problem.

We should be able to publish and get our own data freely.

I suspect that well over half of those articles mentioned Google and not investing in search.

Which is the same argument, just with less techy jargon in it.

Gonna go to a meeting and then rewrite a bunch of examples. ttfn.

For the internet / society to function, information needs to be able to go both top-down and bottom-up.

In a painful nutshell, this is why things are "broken".

Each internet generation has repeated this cycle.

Data opens, people come.
Data closes, people go.

I actually think the path is pretty straight:

Yahoo tries to make a Hierarchical Object Organizer driven by user content, won't allow scraping, won't highlight aggregators, and makes itself useless enough to start to pay Google for search.

Google embraces standards and takes off like a rocket.

I'm actually suggesting a much simpler alternative:

Sites should have low-friction ways to access the data, as well as the text.

This is where #AtProto / xrpc comes in...

See this fun thread from the weekend

bsky.app/profile/mrpo...

I'm not sure that mentioning Yahoo here is a "win"

My point is that sites that attempt to limit scraping limit their own exposure.

Yahoo may have been an "early adopter" of trying to limit scraping.

And that's part of why Yahoo is mostly irrelevant now.

Adding extra salt into these old wounds...

Because imdb content is almost all user-generated, they are built _with our help_.

If you're posting to imdb, you're giving them the right to use your copyright, not giving them your copyright.

It's _our_ data. They just output it.

It's _always_ been that internet.

And that's what's made the internet the internet:

The "Network Effect"

Pulling out pieces like this make a site irrelevant.

All imdb has done here is hastened their own decline.

Additional context here on the legalese:

1. Merely visiting / curling / Invoke-RestMethoding a site does not bind you to their terms of service.

(You generally need to explicitly agree for it to be binding)

2. Asking for a single piece of information has negligible impact and harm.

It's a fundamental underpinning of the internet...

If I cannot request content from your site, it is useless to me.

If only well-heeled companies with millions of dollars can index content, it stops the "network effect" from working.

🤷 Maybe I'm just old, but #BackInMyDay the internet was open.

Yep, it also appears that they broke the validator for schema.org

Still works plenty of other places.

🤷 guess I'm updating some modules today...

Thanks for the repro.

Also, I don't think asking for a single movie counts as "scraping".

I would call it "indexing".

🤷 Just another big company breaking the way the Internet has worked for years.

Also, this worked ~9 months back (when I wrote the examples that I'll now have to switch to LetterBoxed).

Does it work on your machine?

Did it reliably reproduce for you?

Also, in this case, SEO isn't "snake oil", it's a set of standards that most sites adopt for social integration (OpenGraph) and search (JSON-LD).

First of all:

Did it reliably reproduce? (returning nothing for you, too?)

Second of all:

This works on basically all sites, since it's how SEO works.

That's why I'm so shocked at this chicanery.

It's literally worked this way for over a decade.

Who said anything about a rest api?

I'm talking about requesting a page and getting back it's JSON-LD / OpenGraph information.

Just pick a random title's webpage and Invoke-RestMethod

Please sanity check me.

Quite sure. Can access the site just fine. Asking from the terminal literally gives a blank response. Same with curl.

Pick a title, any title, and try to reproduce the issue.

I'd honestly be very happy if it was "just me" (somehow)

Even still failed when providing a different UserAgent 🫤

When did IMDB start providing an empty response to curl / PowerShell / local tools?

😠😡😞🤦

Guess I'm gonna have to switch some examples over to LetterBoxed.

Anyone got a good recommendation for tv databases?

MrPowerShell/MrPowerShell.com/Modules/index.html.ps1 at main · StartAutomating/MrPowerShell Jack of all Trades, Master of PowerShell. Contribute to StartAutomating/MrPowerShell development by creating an account on GitHub.

The display side of things is where things get extra magical.

Because we can now view our site as data, we can easily customize the display of the data.

So the modules endpoint has my modules.

To render the page, we're basically just viewing the objects.

github.com/StartAutomat...

lol was actually hoping arm64 on Linux 😉 . Would love to run this on the #RaspberryPi (cause it would give sleek dedicated device GUIs we could plug into a TV ;-) )

MrPowerShell/MrPowerShell.com/xrpc/com.powershellgallery.modules/index.json.ps1 at main · StartAutomating/MrPowerShell Jack of all Trades, Master of PowerShell. Contribute to StartAutomating/MrPowerShell development by creating an account on GitHub.

Let's look at the json schema side of things first.

Any PowerShell script can define _any_ OutputType, including an arbitrary string.

So we can just go ahead and defined the schema on the script that generates the endpoint.

_and use that schema to massage results_ 😎

github.com/StartAutomat...

My PowerShell Modules My PowerShell modules published to the PowerShell Gallery.

For another example, let's look at My Modules Page:

mrpowershell.com/Modules

It's using data available from the xrpc endpoint:

mrpowershell.com/xrpc/com.pow...

This endpoint is built by taking PowerShell Gallery information and making it adhere to a json schema.

Standard.site - One schema. Every platform. Standard.site provides shared lexicons for long-form publishing on AT Protocol. Making content easier to discover, index, and move across the ATmosphere.

How?

First step, in a static site generator, simply make an index.json file beneath `/xrpc/`

For a trivial example, I've made a standard.site publication endpoint for my own website:

mrpowershell.com/xrpc/site.st...

This lets anyone get right to the data, without a pds.

This weekend has brought a few "new" discoveries:

1. We can make dynamically static xrpc endpoints
2. We can put json schemas in PowerShell OutputTypes
3. We can automatically decorate any XRPC data to customize display
4. We can combined these tricks to easily make data driven websites!

😎

Very cool!

Please add arm64 support when you get the chance...

GitHub - mdgrs-mei/AvaloniaUIShell: Cross-platform Desktop GUI framework for PowerShell powered by Avalonia UI. Cross-platform Desktop GUI framework for PowerShell powered by Avalonia UI. - mdgrs-mei/AvaloniaUIShell

Published a new module called AvaloniaUIShell, (the world's first?) cross-platform desktop GUI framework for PowerShell.

github.com/mdgrs-mei/Av...

It's using the same mechanism as WinUIShell and takes care of all threading and non-blocking UI operations.

#PowerShell

Nifty! I suspect getting the "explorer" view right will be the tricky part, if you ever try to support multiple platforms.

Thanks for addressing the concerns!