Merge branch 'dev' into pr/52

.docs/content/1.get-started/1.quick-start.md

@@ -21,7 +21,7 @@ Let's get started with `@movie-web/providers`. First lets install the package.
 To get started with scraping on the **server**, first you have to make an instance of the providers.
 
 ::alert{type="warning"}
-This snippet will only work on a **server**, for other environments, check out [Usage on X](../2.essentials/0.usage-on-x.md).
+This snippet will only work on a **server**. For other environments, check out [Usage on X](../2.essentials/0.usage-on-x.md).
 ::
 
 ```ts [index.ts (server)]
@@ -39,8 +39,8 @@ const providers = makeProviders({
 })
 ```
 
-Perfect, this instance of the providers you can reuse everywhere where you need to.
+Perfect. You now have an instance of the providers you can reuse everywhere.
-Now lets actually scrape an item:
+Now let's scrape an item:
 
 ```ts [index.ts (server)]
 // fetch some data from TMDB

.docs/content/1.get-started/4.changelog.md

@@ -2,6 +2,40 @@
 title: 'Changelog'
 ---
 
+# Version 2.1.1
+ - Fixed vidplay decryption keys being wrong and switched the domain to one that works
+
+# Version 2.1.0
+ - Add preferedHeaders to most sources
+ - Add CF_BLOCKED flag to sources that have blocked cloudflare API's
+ - Fix vidsrc sometimes having an equal sign where it shouldnt
+ - Increase ranking of lookmovie
+ - Re-enabled subtitles for febbox-mp4 
+
+# Version 2.0.5
+- Disable subtitles for febbox-mp4. As their endpoint doesn't work anymore.
+
+# Version 2.0.4
+- Added providers:
+  - Add VidSrcTo provider with Vidplay and Filemoon embeds
+  - Add VidSrc provider with StreamBucket embeds
+- Fixed providers:
+  - RemoteStream
+  - LookMovie - Fixed captions
+  - ShowBox
+- Updated documentation to fix spelling + grammar
+- User-agent header fix
+- Needs the latest simple-proxy update
+- Added utility to not return multiple subs for the same language - Applies to Lookmovie and Showbox
+
+# Version 2.0.3
+- Actually remove Febbox HLS 
+
+# Version 2.0.2
+- Added Lookmovie caption support
+- Fix Febbox duplicate subtitle languages
+- Remove Febbox HLS
+
 # Version 2.0.1
 - Fixed issue where febbox-mp4 would not show all qualities
 - Fixed issue where discoverEmbeds event would not show the embeds in the right order
@@ -22,7 +56,7 @@ There are breaking changes in this list, make sure to read them thoroughly if yo
 - Fetchers can now return a full response with headers and everything
 
 **New features:**
-- Added system to allow scraping ip locked sources through the consistentIpforRequests option.
+- Added system to allow scraping IP locked sources through the consistentIpforRequests option.
 - There is now a `buildProviders()` function that gives a builder for the `ProviderControls`. It's an alternative to `makeProviders()`.
 - Streams can now return a headers object and a `preferredHeaders` object. which is required and optional headers for when using the stream.
 

.docs/content/2.essentials/0.usage-on-x.md

@@ -2,12 +2,12 @@
 
 The library can run in many environments, so it can be tricky to figure out how to set it up.
 
-So here is a checklist, for more specific environments, keep reading below:
+Here is a checklist. For more specific environments, keep reading below:
  - When requests are very restricted (like browser client-side). Configure a proxied fetcher.
- - When your requests come from the same device it will be streamed on (Not compatible with proxied fetcher). Set `consistentIpForRequests: true`.
+ - When your requests come from the same device on which it will be streamed (not compatible with proxied fetcher). Set `consistentIpForRequests: true`.
  - To set a target. Consult [Targets](./1.targets.md).
 
-To make use of the examples below, You check check out the following pages:
+To make use of the examples below, check out the following pages:
  - [Quick start](../1.get-started/1.quick-start.md)
  - [Using streams](../2.essentials/4.using-streams.md)
 

.docs/content/2.essentials/1.targets.md

@@ -3,7 +3,7 @@
 When creating provider controls, you will immediately be required to choose a target.
 
 ::alert{type="warning"}
-A target is the device where the stream will be played on.
+A target is the device on which the stream will be played.
 **Where the scraping is run has nothing to do with the target**, only where the stream is finally played in the end is significant in choosing a target.
 ::
 

.docs/content/2.essentials/2.fetchers.md

@@ -1,7 +1,7 @@
 # Fetchers
 
-When creating provider controls, it will need you to configure a fetcher.
+When creating provider controls, a fetcher will need to be configured.
-This comes with some considerations depending on the environment youre running.
+Depending on your environment, this can come with some considerations:
 
 ## Using `fetch()`
 In most cases, you can use the `fetch()` API. This will work in newer versions of Node.js (18 and above) and on the browser.
@@ -19,7 +19,7 @@ const fetcher = makeStandardFetcher(fetch);
 ```
 
 ## Using fetchers on the browser
-When using this library on a browser, you will need a proxy. Browsers come with many restrictions on when a web request can be made, and to bypass those restrictions, you will need a cors proxy.
+When using this library on a browser, you will need a proxy. Browsers restrict when a web request can be made. To bypass those restrictions, you will need a CORS proxy.
 
 The movie-web team has a proxy pre-made and pre-configured for you to use. For more information, check out [movie-web/simple-proxy](https://github.com/movie-web/simple-proxy). After installing, you can use this proxy like so:
 
@@ -31,13 +31,13 @@ If you aren't able to use this specific proxy and need to use a different one, y
 
 ## Making a derived fetcher
 
-In some rare cases, a custom fetcher will need to be made. This can be quite difficult to do from scratch so it's recommended to base it off an existing fetcher and building your own functionality around it.
+In some rare cases, a custom fetcher is necessary. This can be quite difficult to make from scratch so it's recommended to base it off of an existing fetcher and building your own functionality around it.
 
 ```ts
 export function makeCustomFetcher(): Fetcher {
   const fetcher = makeStandardFetcher(f);
   const customFetcher: Fetcher = (url, ops) => {
-    // Do something with the options and url here
+    // Do something with the options and URL here
     return fetcher(url, ops);
   };
 
@@ -45,19 +45,19 @@ export function makeCustomFetcher(): Fetcher {
 }
 ```
 
-If you need to make your own fetcher for a proxy. Make sure you make it compatible with the following headers: `Set-Cookie`, `Cookie`, `Referer`, `Origin`. Proxied fetchers need to be able to write/read those headers when making a request.
+If you need to make your own fetcher for a proxy, ensure you make it compatible with the following headers: `Set-Cookie`, `Cookie`, `Referer`, `Origin`. Proxied fetchers need to be able to write/read those headers when making a request.
 
 
 ## Making a fetcher from scratch
 
-In some even rare cases, you need to make one completely from scratch.
+In some rare cases, you need to make a fetcher from scratch.
 This is the list of features it needs:
  - Send/read every header
  - Parse JSON, otherwise parse as text
  - Send JSON, Formdata or normal strings
- - get final destination url
+ - get final destination URL
 
-It's not recommended to do this at all, but if you have to. You can base your code on the original implementation of `makeStandardFetcher`. Check the out [source code for it here](https://github.com/movie-web/providers/blob/dev/src/fetchers/standardFetch.ts).
+It's not recommended to do this at all. If you have to, you can base your code on the original implementation of `makeStandardFetcher`. Check out the [source code for it here](https://github.com/movie-web/providers/blob/dev/src/fetchers/standardFetch.ts).
 
 Here is a basic template on how to make your own custom fetcher:
 

.docs/content/2.essentials/3.customize-providers.md

@@ -1,6 +1,6 @@
 # Customize providers
 
-You make a provider controls in two ways. Either with `makeProviders()` (the simpler option) or with `buildProviders()` (more elaborate and extensive option).
+You make the provider controls in two ways. Either with `makeProviders()` (the simpler option) or with `buildProviders()` (more elaborate and extensive option).
 
 ## `makeProviders()` (simple)
 
@@ -41,7 +41,7 @@ const providers = buildProviders()
 
 ### Adding only select few providers
 
-Not all providers are great quality, so you can make a instance of the controls with only the providers you want.
+Not all providers are great quality, so you can make an instance of the controls with only the providers you want.
 
 ```ts
 const providers = buildProviders()
@@ -55,7 +55,7 @@ const providers = buildProviders()
 
 ### Adding your own scrapers to the providers
 
-If you have your own scraper and still want to use the nice utils of the provider library or just want to add on to the builtin providers. You can add your own custom source.
+If you have your own scraper and still want to use the nice utilities of the provider library or just want to add on to the built-in providers, you can add your own custom source.
 
 ```ts
 const providers = buildProviders()

.docs/content/2.essentials/4.using-streams.md

@@ -12,11 +12,11 @@ All streams have the same common parameters:
  - `Stream.headers`: Either undefined or a key value object of headers you must set to use the stream.
  - `Stream.preferredHeaders`: Either undefined or a key value object of headers you may want to set if you want optimal playback - but not required.
 
-Now let's delve deeper into how to actually watch these streams!
+Now let's delve deeper into how to watch these streams!
 
 ## Streams with type `hls`
 
-HLS streams can be tough to watch, it's not a normal file you can just use.
+HLS streams can be tough to watch. They're not normal files you can just use.
 These streams have an extra property `Stream.playlist` which contains the m3u8 playlist.
 
 Here is a code sample of how to use HLS streams in web context using hls.js
@@ -39,17 +39,17 @@ Here is a code sample of how to use HLS streams in web context using hls.js
 
 ## Streams with type `file`
 
-File streams are quite easy to use, it just returns a new property: `Stream.qualities`.
+File streams are quite easy to use, they just return a new property: `Stream.qualities`.
-This property is a map of quality and a stream file. So if you want to get 1080p quality you do `stream["1080"]` to get your stream file. It will return undefined if there is no quality like that.
+This property is a map of quality and a stream file. So if you want to get 1080p quality you do `stream["1080"]` to get your stream file. It will return undefined if that quality is absent.
 
 The possibly qualities are: `unknown`, `360`, `480`, `720`, `1080`, `4k`.
-File based streams are garuanteed to always have one quality.
+File based streams are always guaranteed to have one quality.
 
 Once you get a streamfile, you have the following parameters:
  - `StreamFile.type`: Right now it can only be `mp4`.
  - `StreamFile.url`: The URL linking to the video file.
 
-Here is a code sample of how to watch a file based stream the video in a browser:
+Here is a code sample of how to watch a file based stream in a browser:
 
 ```html
 <video id="video"></video>
@@ -66,9 +66,9 @@ Here is a code sample of how to watch a file based stream the video in a browser
 ## Streams with headers
 
 Streams have both a `Stream.headers` and a `Stream.preferredHeaders`.
-The difference between the two is that `Stream.headers` **must** be set in other for the stream to work. While the other one is optional, and can only enhance the quality or performance.
+The difference between the two is that `Stream.headers` **must** be set in order for the stream to work. While the other is optional, and enhances the quality or performance.
 
-If your target is set to `BROWSER`. There will never be required headers, as it's not possible to do.
+If your target is set to `BROWSER`, headers will never be required, as it's not possible to do.
 
 ## Using captions/subtitles
 
@@ -77,7 +77,7 @@ All streams have a list of captions at `Stream.captions`. The structure looks li
 type Caption = {
   type: CaptionType; // Language type, either "srt" or "vtt"
   id: string; // Unique per stream
-  url: string; // The url pointing to the subtitle file
+  url: string; // The URL pointing to the subtitle file
   hasCorsRestrictions: boolean; // If true, you will need to proxy it if you're running in a browser
   language: string; // Language code of the caption
 };

.docs/content/3.in-depth/2.flags.md

@@ -1,10 +1,10 @@
 # Flags
 
-Flags is the primary way the library seperates entities between different environments.
+Flags is the primary way the library separates entities between different environments.
-For example some sources only give back content that has the CORS headers set to allow anyone, so that source gets the flag `CORS_ALLOWED`. Now if you set your target to `BROWSER`, sources without that flag won't even get listed.
+For example, some sources only give back content that has the CORS headers set to allow anyone, so that source gets the flag `CORS_ALLOWED`. Now if you set your target to `BROWSER`, sources without that flag won't even get listed.
 
 This concept is applied in multiple away across the library.
 
 ## Flag options
  - `CORS_ALLOWED`: Headers from the output streams are set to allow any origin.
- - `IP_LOCKED`: The streams are locked by ip, requester and watcher must be the same.
+ - `IP_LOCKED`: The streams are locked by IP: requester and watcher must be the same.

.docs/content/4.extra-topics/0.development.md

@@ -15,13 +15,13 @@ TODO
 ## Testing using the CLI
 
 Testing can be quite difficult for this library, unit tests can't really be made because of the unreliable nature of scrapers.
-But manually testing by writing an entrypoint is also really annoying.
+But manually testing by writing an entry-point is also really annoying.
 
-Our solution is to make a CLI that you can use to run the scrapers, for everything else there are unit tests.
+Our solution is to make a CLI that you can use to run the scrapers. For everything else there are unit tests.
 
 ### Setup
-Make a `.env` file in the root of the repository and add a TMDB api key: `MOVIE_WEB_TMDB_API_KEY=KEY_HERE`.
+Make a `.env` file in the root of the repository and add a TMDB API key: `MOVIE_WEB_TMDB_API_KEY=KEY_HERE`.
-Then make sure you've ran `npm i` to get all the dependencies.
+Then make sure you've run `npm i` to get all the dependencies.
 
 ### Mode 1 - interactive
 

.docs/content/5.api-reference/0.makeProviders.md

@@ -1,7 +1,7 @@
 # `makeProviders`
 
 Make an instance of provider controls with configuration. 
-This is the main entrypoint of the library. It is recommended to make one instance globally and reuse it throughout your application.
+This is the main entry-point of the library. It is recommended to make one instance globally and reuse it throughout your application.
 
 ## Example
 
@@ -23,9 +23,9 @@ interface ProviderBuilderOptions {
   // instance of a fetcher, all webrequests are made with the fetcher.
   fetcher: Fetcher;
   
-  // instance of a fetcher, in case the request has cors restrictions.
+  // instance of a fetcher, in case the request has CORS restrictions.
   // this fetcher will be called instead of normal fetcher.
-  // if your environment doesnt have cors restrictions (like Node.JS), there is no need to set this.
+  // if your environment doesn't have CORS restrictions (like Node.JS), there is no need to set this.
   proxiedFetcher?: Fetcher;
 
   // target to get streams for

.docs/content/5.api-reference/1.ProviderControlsRunAll.md

@@ -1,7 +1,7 @@
 # `ProviderControls.runAll`
 
 Run all providers one by one in order of their built-in ranking.
-You can attach events if you need to know what is going on while its processing.
+You can attach events if you need to know what is going on while it is processing.
 
 ## Example
 
@@ -20,7 +20,7 @@ const stream = await providers.runAll({
 })
 
 // scrape a stream, but prioritize flixhq above all
-// (other scrapers are stil ran if flixhq fails, it just has priority)
+// (other scrapers are still run if flixhq fails, it just has priority)
 const flixhqStream = await providers.runAll({
   media: media,
   sourceOrder: ['flixhq']
@@ -33,12 +33,12 @@ const flixhqStream = await providers.runAll({
 function runAll(runnerOps: RunnerOptions): Promise<RunOutput | null>;
 
 interface RunnerOptions {
-  // overwrite the order of sources to run. list of ids
+  // overwrite the order of sources to run. List of IDs
-  // any omitted ids are in added to the end in order of rank (highest first)
+  // any omitted IDs are added to the end in order of rank (highest first)
   sourceOrder?: string[];
 
-  // overwrite the order of embeds to run. list of ids
+  // overwrite the order of embeds to run. List of IDs
-  // any omitted ids are in added to the end in order of rank (highest first)
+  // any omitted IDs are added to the end in order of rank (highest first)
   embedOrder?: string[];
 
   // object of event functions
@@ -49,13 +49,13 @@ interface RunnerOptions {
 }
 
 type RunOutput = {
-  // source scraper id
+  // source scraper ID
   sourceId: string;
 
-  // if from an embed, this is the embed scraper id
+  // if from an embed, this is the embed scraper ID
   embedId?: string;
 
-  // the outputed stream
+  // the emitted stream
   stream: Stream;
 };
 ```

.docs/content/5.api-reference/2.ProviderControlsrunSourceScraper.md

@@ -1,6 +1,6 @@
 # `ProviderControls.runSourceScraper`
 
-Run a specific source scraper and get its outputted streams.
+Run a specific source scraper and get its emitted streams.
 
 ## Example
 
@@ -24,7 +24,7 @@ try {
   })
 } catch (err) {
   if (err instanceof NotFoundError) {
-    console.log('source doesnt have this media');
+    console.log('source does not have this media');
   } else {
     console.log('failed to scrape')
   }
@@ -48,13 +48,13 @@ interface SourceRunnerOptions {
   // the media you want to see sources from
   media: ScrapeMedia;
 
-  // id of the source scraper you want to scrape from
+  // ID of the source scraper you want to scrape from
   id: string;
 }
 
 type SourcererOutput = {
   // list of embeds that the source scraper found.
-  // embed id is a reference to an embed scraper
+  // embed ID is a reference to an embed scraper
   embeds: {
     embedId: string;
     url: string;

.docs/content/5.api-reference/3.ProviderControlsrunEmbedScraper.md

@@ -1,6 +1,6 @@
 # `ProviderControls.runEmbedScraper`
 
-Run a specific embed scraper and get its outputted streams.
+Run a specific embed scraper and get its emitted streams.
 
 ## Example
 
@@ -31,10 +31,10 @@ interface EmbedRunnerOptions {
   // object of event functions
   events?: IndividualScraperEvents;
 
-  // the embed url
+  // the embed URL
   url: string;
 
-  // id of the embed scraper you want to scrape from
+  // ID of the embed scraper you want to scrape from
   id: string;
 }
 

.docs/content/5.api-reference/4.ProviderControlslistSources.md

@@ -1,13 +1,13 @@
 # `ProviderControls.listSources`
 
-List all source scrapers that applicable for the target.
+List all source scrapers that are applicable for the target.
-They are sorted by rank, highest first
+They are sorted by rank; highest first
 
 ## Example
 
 ```ts
 const sourceScrapers = providers.listSources();
-// Guaranteed to only return type: 'source'
+// Guaranteed to only return the type: 'source'
 ```
 
 ## Type

.docs/content/5.api-reference/5.ProviderControlslistEmbeds.md

@@ -1,13 +1,13 @@
 # `ProviderControls.listEmbeds`
 
-List all embed scrapers that applicable for the target.
+List all embed scrapers that are applicable for the target.
-They are sorted by rank, highest first
+They are sorted by rank; highest first
 
 ## Example
 
 ```ts
 const embedScrapers = providers.listEmbeds();
-// Guaranteed to only return type: 'embed'
+// Guaranteed to only return the type: 'embed'
 ```
 
 ## Type

.docs/content/5.api-reference/6.ProviderControlsgetMetadata.md

@@ -1,7 +1,7 @@
 # `ProviderControls.getMetadata`
 
 Get meta data for a scraper, can be either source or embed scraper.
-Returns null if the `id` is not recognized.
+Returns `null` if the `id` is not recognized.
 
 ## Example
 

.docs/content/5.api-reference/7.makeStandardFetcher.md

@@ -1,6 +1,6 @@
 # `makeStandardFetcher`
 
-Make a fetcher from a `fetch()` API. It is used for making a instance of provider controls.
+Make a fetcher from a `fetch()` API. It is used for making an instance of provider controls.
 
 ## Example
 

.docs/package-lock.json

@@ -6474,9 +6474,9 @@
       }
     },
     "node_modules/follow-redirects": {
-      "version": "1.15.2",
+      "version": "1.15.4",
-      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.2.tgz",
+      "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.15.4.tgz",
-      "integrity": "sha512-VQLG33o04KaQ8uYi2tVNbdrWp1QWxNNea+nmIB4EVM28v0hmP17z7aG1+wAkNzVq4KeXTq3221ye5qTJP91JwA==",
+      "integrity": "sha512-Cr4D/5wlrb0z9dgERpUL3LrmPKVDsETIJhaCMeDfuFYcqa5bldGV6wBsAN6X/vxlXQtFBMrXdXxdL8CbDTGniw==",
       "dev": true,
       "funding": [
         {
@@ -17286,9 +17286,9 @@
       "dev": true
     },
     "node_modules/vite": {
-      "version": "4.5.1",
+      "version": "4.5.2",
-      "resolved": "https://registry.npmjs.org/vite/-/vite-4.5.1.tgz",
+      "resolved": "https://registry.npmjs.org/vite/-/vite-4.5.2.tgz",
-      "integrity": "sha512-AXXFaAJ8yebyqzoNB9fu2pHoo/nWX+xZlaRwoeYUxEqBO+Zj4msE5G+BhGBll9lYEKv9Hfks52PAF2X7qDYXQA==",
+      "integrity": "sha512-tBCZBNSBbHQkaGyhGCDUGqeo2ph8Fstyp6FMSvTtsXeZSPpSMGlviAOav2hxVTqFcx8Hj/twtWKsMJXNY0xI8w==",
       "dev": true,
       "dependencies": {
         "esbuild": "^0.18.10",

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "@movie-web/providers",
-  "version": "1.1.5",
+  "version": "2.1.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "@movie-web/providers",
-      "version": "1.1.5",
+      "version": "2.1.0",
       "license": "MIT",
       "dependencies": {
         "cheerio": "^1.0.0-rc.12",
@@ -26,6 +26,7 @@
         "@typescript-eslint/parser": "^5.60.0",
         "@vitest/coverage-v8": "^0.34.3",
         "commander": "^11.0.0",
+        "cross-env": "^7.0.3",
         "dotenv": "^16.3.1",
         "enquirer": "^2.4.1",
         "eslint": "^8.30.0",
@@ -1794,6 +1795,24 @@
       "dev": true,
       "license": "MIT"
     },
+    "node_modules/cross-env": {
+      "version": "7.0.3",
+      "resolved": "https://registry.npmjs.org/cross-env/-/cross-env-7.0.3.tgz",
+      "integrity": "sha512-+/HKd6EgcQCJGh2PSjZuUitQBQynKor4wrFbRg4DtAgS1aWO+gU52xpH7M9ScGgXSYmAVS9bIJ8EzuaGw0oNAw==",
+      "dev": true,
+      "dependencies": {
+        "cross-spawn": "^7.0.1"
+      },
+      "bin": {
+        "cross-env": "src/bin/cross-env.js",
+        "cross-env-shell": "src/bin/cross-env-shell.js"
+      },
+      "engines": {
+        "node": ">=10.14",
+        "npm": ">=6",
+        "yarn": ">=1"
+      }
+    },
     "node_modules/cross-fetch": {
       "version": "4.0.0",
       "resolved": "https://registry.npmjs.org/cross-fetch/-/cross-fetch-4.0.0.tgz",
@@ -2848,6 +2867,20 @@
       "dev": true,
       "license": "ISC"
     },
+    "node_modules/fsevents": {
+      "version": "2.3.3",
+      "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
+      "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
+      "dev": true,
+      "hasInstallScript": true,
+      "optional": true,
+      "os": [
+        "darwin"
+      ],
+      "engines": {
+        "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
+      }
+    },
     "node_modules/function-bind": {
       "version": "1.1.2",
       "dev": true,
@@ -5517,9 +5550,10 @@
       }
     },
     "node_modules/vite": {
-      "version": "4.5.1",
+      "version": "4.5.2",
+      "resolved": "https://registry.npmjs.org/vite/-/vite-4.5.2.tgz",
+      "integrity": "sha512-tBCZBNSBbHQkaGyhGCDUGqeo2ph8Fstyp6FMSvTtsXeZSPpSMGlviAOav2hxVTqFcx8Hj/twtWKsMJXNY0xI8w==",
       "dev": true,
-      "license": "MIT",
       "dependencies": {
         "esbuild": "^0.18.10",
         "postcss": "^8.4.27",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@movie-web/providers",
-  "version": "2.0.3",
+  "version": "2.1.1",
   "description": "Package that contains all the providers of movie-web",
   "main": "./lib/index.umd.js",
   "types": "./lib/index.d.ts",
@@ -38,6 +38,7 @@
     "cli": "ts-node ./src/dev-cli/index.ts",
     "test": "vitest run",
     "test:watch": "vitest",
+    "test:providers": "cross-env MW_TEST_PROVIDERS=true vitest run --reporter verbose",
     "test:integration": "node ./tests/cjs && node ./tests/esm && node ./tests/browser",
     "test:coverage": "vitest run --coverage",
     "lint": "eslint --ext .ts,.js src/",
@@ -55,6 +56,7 @@
     "@typescript-eslint/parser": "^5.60.0",
     "@vitest/coverage-v8": "^0.34.3",
     "commander": "^11.0.0",
+    "cross-env": "^7.0.3",
     "dotenv": "^16.3.1",
     "enquirer": "^2.4.1",
     "eslint": "^8.30.0",

src/__test__/providers/embedUtils.ts

@@ -0,0 +1,90 @@
+import { buildProviders } from "@/entrypoint/builder";
+import { ScrapeMedia } from "@/entrypoint/utils/media";
+import { targets } from "@/entrypoint/utils/targets";
+import { makeStandardFetcher } from "@/fetchers/standardFetch";
+import { Embed, Sourcerer, SourcererEmbed } from "@/providers/base";
+import { TestTypes } from "./providerUtils";
+import { describe, expect, it } from "vitest";
+import { ProviderControls } from "@/entrypoint/controls";
+import { makeSimpleProxyFetcher } from "@/fetchers/simpleProxy";
+
+export interface TestEmbedOptions {
+  embed: Embed;
+  source: Sourcerer;
+  testSuite: ScrapeMedia[];
+  types: TestTypes[];
+  debug?: boolean;
+  expect: {
+    embeds: number;
+    streams?: number;
+    error?: boolean;
+  }
+}
+
+function makeBaseEmbedProviders() {
+  const builder = buildProviders()
+    .setTarget(targets.ANY)
+    .setFetcher(makeStandardFetcher(fetch));
+  return builder;
+}
+
+export function testEmbed(ops: TestEmbedOptions) {
+  if (ops.testSuite.length === 0) throw new Error("Test suite must have at least one test");
+  describe(`embed:${ops.source.id}:${ops.embed.id}`, () => {
+    ops.testSuite.forEach((test) => {
+      describe(`test ${test.title}`, async () => {
+        async function gatherEmbeds(providers: ProviderControls): Promise<SourcererEmbed[]> {
+          const results = await providers.runSourceScraper({
+            id: ops.source.id,
+            media: test,
+          })
+          if (results.embeds.length !== ops.expect.embeds) throw new Error(`Embeds don't match expected amount of embeds (${ops.source.id}, ${ops.embed.id}, got ${results.embeds.length} but expected ${ops.expect.embeds})`);
+          return results.embeds;
+        }
+
+        async function runTest(providers: ProviderControls, embedUrl: string) {
+          let hasError = false;
+          let streamCount = 0;
+          try {
+            const result = await providers.runEmbedScraper({
+              id: ops.embed.id,
+              url: embedUrl,
+            })
+            if (ops.debug) console.log(result);
+            streamCount = (result.stream ?? []).length;
+          } catch (err) {
+            if (ops.debug) console.log(err);
+            hasError = true;
+          }
+          expect(ops.expect.error ?? false).toBe(hasError);
+          expect(ops.expect.streams ?? 0).toBe(streamCount);
+        }
+
+        for (const t of ops.types) {
+          const builder = makeBaseEmbedProviders().addSource(ops.source).addEmbed(ops.embed);
+          if (t === 'standard') {}
+          else if (t === 'ip:standard')
+            builder.enableConsistentIpForRequests();
+          else if (t === 'proxied') {
+            if (!process.env.MOVIE_WEB_PROXY_URL)
+              throw new Error("Cant use proxied test without setting MOVIE_WEB_PROXY_URL env");
+            builder.setProxiedFetcher(makeSimpleProxyFetcher(process.env.MOVIE_WEB_PROXY_URL, fetch));
+          }
+          const providers = builder.build();
+          try {
+            const embeds = await gatherEmbeds(providers);
+            embeds.forEach((embed, i) => {
+              it(`${t} - embed ${i}`, async () => {
+                await runTest(providers, embed.url);
+              })
+            })
+          } catch (err) {
+            it(`${t} - embed ??`, () => {
+              throw new Error("Failed to get streams: " + err);
+            })
+          }
+        }
+      })
+    })
+  })
+}

src/__test__/providers/embeds.test.ts

@@ -0,0 +1,118 @@
+import dotenv from 'dotenv';
+import { febboxMp4Scraper } from "@/providers/embeds/febbox/mp4";
+import { testEmbed } from "./embedUtils";
+import { showboxScraper } from "@/providers/sources/showbox";
+import { testMedia } from "./testMedia";
+import { flixhqScraper } from "@/providers/sources/flixhq";
+import { upcloudScraper } from "@/providers/embeds/upcloud";
+import { goMoviesScraper } from "@/providers/sources/gomovies";
+import { smashyStreamScraper } from "@/providers/sources/smashystream";
+import { smashyStreamDScraper } from "@/providers/embeds/smashystream/dued";
+import { vidsrcembedScraper } from '@/providers/embeds/vidsrc';
+import { vidsrcScraper } from '@/providers/sources/vidsrc';
+import { vidSrcToScraper } from '@/providers/sources/vidsrcto';
+import { vidplayScraper } from '@/providers/embeds/vidplay';
+import { fileMoonScraper } from '@/providers/embeds/filemoon';
+import { zoechipScraper } from '@/providers/sources/zoechip';
+import { mixdropScraper } from '@/providers/embeds/mixdrop';
+
+dotenv.config();
+
+testEmbed({
+  embed: febboxMp4Scraper,
+  source: showboxScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: upcloudScraper,
+  source: flixhqScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: upcloudScraper,
+  source: goMoviesScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: smashyStreamDScraper,
+  source: smashyStreamScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: vidsrcembedScraper,
+  source: vidsrcScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: vidplayScraper,
+  source: vidSrcToScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: fileMoonScraper,
+  source: vidSrcToScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: upcloudScraper,
+  source: zoechipScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 2,
+    streams: 1,
+  }
+})
+
+testEmbed({
+  embed: mixdropScraper,
+  source: zoechipScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 2,
+    streams: 1,
+  }
+})

src/__test__/providers/providerUtils.ts

@@ -0,0 +1,102 @@
+import { ScrapeMedia } from "@/entrypoint/utils/media";
+import { Embed, Sourcerer, SourcererEmbed } from "@/providers/base";
+import { buildProviders } from "@/entrypoint/builder";
+import { describe, expect, it } from "vitest";
+import { makeStandardFetcher } from "@/fetchers/standardFetch";
+import { ProviderControls } from "@/entrypoint/controls";
+import { NotFoundError } from "@/utils/errors";
+import { targets } from "@/entrypoint/utils/targets";
+import { getBuiltinEmbeds } from "@/entrypoint/providers";
+import { makeSimpleProxyFetcher } from "@/fetchers/simpleProxy";
+
+export type TestTypes = 'standard' | 'ip:standard' | 'proxied';
+
+export interface TestSourceOptions {
+  source: Sourcerer;
+  testSuite: ScrapeMedia[];
+  types: TestTypes[];
+  debug?: boolean;
+  expect: {
+    embeds?: number;
+    streams?: number;
+    error?: boolean;
+    notfound?: boolean;
+  }
+}
+
+function makeBaseProviders() {
+  const builder = buildProviders()
+    .setTarget(targets.ANY)
+    .setFetcher(makeStandardFetcher(fetch));
+  const embeds = getBuiltinEmbeds();
+  embeds.forEach(embed => builder.addEmbed(embed));
+  return builder;
+}
+
+export function testSource(ops: TestSourceOptions) {
+  if (ops.testSuite.length === 0) throw new Error("Test suite must have at least one test");
+  describe(`source:${ops.source.id}`, () => {
+    ops.testSuite.forEach((test) => {
+      describe(`test ${test.title}`, () => {
+        async function runTest(providers: ProviderControls) {
+          let hasNotFound = false;
+          let hasError = false;
+          let streamCount = 0;
+          let embedCount = 0;
+          let embeds = [];
+          try {
+            const result = await providers.runSourceScraper({
+              id: ops.source.id,
+              media: test,
+            })
+            if (ops.debug) console.log(result);
+            streamCount = (result.stream ?? []).length;
+            embedCount = result.embeds.length;
+          } catch (err) {
+            if (ops.debug) console.log(err);
+            if (err instanceof NotFoundError)
+              hasNotFound = true;
+            else
+              hasError = true;
+          }
+          expect(ops.expect.error ?? false).toBe(hasError);
+          expect(ops.expect.notfound ?? false).toBe(hasNotFound);
+          expect(ops.expect.streams ?? 0).toBe(streamCount);
+          expect(ops.expect.embeds ?? 0).toBe(embedCount);
+        }
+
+        if (ops.types.includes('standard')) {
+          it(`standard`, async () => {
+            const providers = makeBaseProviders()
+              .addSource(ops.source)
+              .build();
+            await runTest(providers);
+          })
+        }
+
+        if (ops.types.includes('ip:standard')) {
+          it(`standard:ip`, async () => {
+            const providers = makeBaseProviders()
+              .addSource(ops.source)
+              .enableConsistentIpForRequests()
+              .build();
+            await runTest(providers);
+          })
+        }
+
+        if (ops.types.includes('proxied')) {
+          it(`proxied`, async () => {
+            if (!process.env.MOVIE_WEB_PROXY_URL)
+              throw new Error("Cant use proxied test without setting MOVIE_WEB_PROXY_URL env");
+            const providers = makeBaseProviders()
+              .addSource(ops.source)
+              .setProxiedFetcher(makeSimpleProxyFetcher(process.env.MOVIE_WEB_PROXY_URL, fetch))
+              .build();
+            await runTest(providers);
+          })
+        }
+                
+      })
+    })
+  })
+}

src/__test__/providers/providers.test.ts

@@ -0,0 +1,95 @@
+import { testSource } from "./providerUtils";
+import { lookmovieScraper } from "@/providers/sources/lookmovie";
+import { testMedia } from "./testMedia";
+import { showboxScraper } from "@/providers/sources/showbox";
+import dotenv from 'dotenv';
+import { flixhqScraper } from "@/providers/sources/flixhq";
+import { goMoviesScraper } from "@/providers/sources/gomovies";
+import { smashyStreamScraper } from "@/providers/sources/smashystream";
+import { vidsrcScraper } from "@/providers/sources/vidsrc";
+import { vidSrcToScraper } from "@/providers/sources/vidsrcto";
+import { zoechipScraper } from "@/providers/sources/zoechip";
+import { remotestreamScraper } from "@/providers/sources/remotestream";
+
+dotenv.config();
+
+testSource({
+  source: lookmovieScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['ip:standard'],
+  expect: {
+    streams: 1,
+  }
+})
+
+testSource({
+  source: showboxScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+  }
+})
+
+testSource({
+  source: flixhqScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+  }
+})
+
+testSource({
+  source: goMoviesScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+  }
+})
+
+testSource({
+  source: smashyStreamScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+  }
+})
+
+testSource({
+  source: vidsrcScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 1,
+  }
+})
+
+testSource({
+  source: vidSrcToScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 2,
+  }
+})
+
+testSource({
+  source: zoechipScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    embeds: 3,
+  }
+})
+
+testSource({
+  source: remotestreamScraper,
+  testSuite: [testMedia.arcane, testMedia.hamilton],
+  types: ['standard', 'proxied'],
+  expect: {
+    streams: 1,
+  }
+})

src/__test__/providers/testMedia.ts

@@ -0,0 +1,30 @@
+import { ScrapeMedia } from "@/entrypoint/utils/media";
+
+function makeMedia(media: ScrapeMedia): ScrapeMedia {
+  return media;
+}
+
+export const testMedia = {
+  arcane: makeMedia({
+    type: "show",
+    title: "Arcane",
+    tmdbId: "94605",
+    releaseYear: 2021,
+    episode: {
+      number: 1,
+      tmdbId: '1953812',
+    },
+    season: {
+      number: 1,
+      tmdbId: '134187',
+    },
+    imdbId: 'tt11126994'
+  }),
+  hamilton: makeMedia({
+    type: 'movie',
+    tmdbId: '556574',
+    imdbId: 'tt8503618',
+    releaseYear: 2020,
+    title: 'Hamilton'
+  })
+}

src/__test__/standard/providerTests.ts

@@ -2,7 +2,7 @@
 import { vi } from 'vitest';
 
 import { gatherAllEmbeds, gatherAllSources } from '@/providers/all';
-import { Embed, Sourcerer } from '@/providers/base';
+import { makeEmbed, makeSourcerer } from '@/providers/base';
 
 export function makeProviderMocks() {
   const embedsMock = vi.fn<Parameters<typeof gatherAllEmbeds>, ReturnType<typeof gatherAllEmbeds>>();
@@ -13,104 +13,104 @@ export function makeProviderMocks() {
   };
 }
 
-const sourceA = {
+const sourceA = makeSourcerer({
   id: 'a',
   name: 'A',
   rank: 1,
   disabled: false,
   flags: [],
-} as Sourcerer;
+});
-const sourceB = {
+const sourceB = makeSourcerer({
   id: 'b',
   name: 'B',
   rank: 2,
   disabled: false,
   flags: [],
-} as Sourcerer;
+});
-const sourceCDisabled = {
+const sourceCDisabled = makeSourcerer({
   id: 'c',
   name: 'C',
   rank: 3,
   disabled: true,
   flags: [],
-} as Sourcerer;
+});
-const sourceAHigherRank = {
+const sourceAHigherRank = makeSourcerer({
   id: 'a',
   name: 'A',
   rank: 100,
   disabled: false,
   flags: [],
-} as Sourcerer;
+});
-const sourceGSameRankAsA = {
+const sourceGSameRankAsA = makeSourcerer({
   id: 'g',
   name: 'G',
   rank: 1,
   disabled: false,
   flags: [],
-} as Sourcerer;
+});
-const fullSourceYMovie = {
+const fullSourceYMovie = makeSourcerer({
   id: 'y',
   name: 'Y',
   rank: 105,
   scrapeMovie: vi.fn(),
   flags: [],
-} as Sourcerer;
+});
-const fullSourceYShow = {
+const fullSourceYShow = makeSourcerer({
   id: 'y',
   name: 'Y',
   rank: 105,
   scrapeShow: vi.fn(),
   flags: [],
-} as Sourcerer;
+});
-const fullSourceZBoth = {
+const fullSourceZBoth = makeSourcerer({
   id: 'z',
   name: 'Z',
   rank: 106,
   scrapeMovie: vi.fn(),
   scrapeShow: vi.fn(),
   flags: [],
-} as Sourcerer;
+});
 
-const embedD = {
+const embedD = makeEmbed({
   id: 'd',
   rank: 4,
   disabled: false,
-} as Embed;
+} as any);
-const embedA = {
+const embedA = makeEmbed({
   id: 'a',
   rank: 5,
   disabled: false,
-} as Embed;
+} as any);
-const embedEDisabled = {
+const embedEDisabled = makeEmbed({
   id: 'e',
   rank: 6,
   disabled: true,
-} as Embed;
+} as any);
-const embedDHigherRank = {
+const embedDHigherRank = makeEmbed({
   id: 'd',
   rank: 4000,
   disabled: false,
-} as Embed;
+} as any);
-const embedFSameRankAsA = {
+const embedFSameRankAsA = makeEmbed({
   id: 'f',
   rank: 5,
   disabled: false,
-} as Embed;
+} as any);
-const embedHSameRankAsSourceA = {
+const embedHSameRankAsSourceA = makeEmbed({
   id: 'h',
   rank: 1,
   disabled: false,
-} as Embed;
+} as any);
-const fullEmbedX = {
+const fullEmbedX = makeEmbed({
   id: 'x',
   name: 'X',
   rank: 104,
-} as Embed;
+} as any);
-const fullEmbedZ = {
+const fullEmbedZ = makeEmbed({
   id: 'z',
   name: 'Z',
   rank: 109,
-} as Embed;
+} as any);
 
 export const mockSources = {
   sourceA,

src/__test__/standard/providers/checks.test.ts

@@ -1,4 +1,4 @@
-import { mockEmbeds, mockSources } from '@/__test__/providerTests';
+import { mockEmbeds, mockSources } from '../providerTests';
 import { getBuiltinEmbeds, getBuiltinSources } from '@/entrypoint/providers';
 import { FeatureMap } from '@/entrypoint/utils/targets';
 import { getProviders } from '@/providers/get';

src/__test__/standard/runner/list.test.ts

@@ -1,4 +1,4 @@
-import { mockEmbeds, mockSources } from '@/__test__/providerTests';
+import { mockEmbeds, mockSources } from '../providerTests.ts';
 import { makeProviders } from '@/entrypoint/declare';
 import { targets } from '@/entrypoint/utils/targets';
 import { afterEach, describe, expect, it, vi } from 'vitest';

src/__test__/standard/runner/meta.test.ts

@@ -1,4 +1,4 @@
-import { mockEmbeds, mockSources } from '@/__test__/providerTests';
+import { mockEmbeds, mockSources } from '../providerTests.ts';
 import { makeProviders } from '@/entrypoint/declare';
 import { targets } from '@/entrypoint/utils/targets';
 import { afterEach, describe, expect, it, vi } from 'vitest';

src/dev-cli/tmdb.ts

@@ -2,7 +2,7 @@ import { getConfig } from '@/dev-cli/config';
 
 import { MovieMedia, ShowMedia } from '..';
 
-export async function makeTMDBRequest(url: string): Promise<Response> {
+export async function makeTMDBRequest(url: string, appendToResponse?: string): Promise<Response> {
   const headers: {
     accept: 'application/json';
     authorization?: string;
@@ -10,7 +10,7 @@ export async function makeTMDBRequest(url: string): Promise<Response> {
     accept: 'application/json',
   };
 
-  let requestURL = url;
+  const requestURL = new URL(url);
   const key = getConfig().tmdbApiKey;
 
   // * JWT keys always start with ey and are ONLY valid as a header.
@@ -19,7 +19,11 @@ export async function makeTMDBRequest(url: string): Promise<Response> {
   if (key.startsWith('ey')) {
     headers.authorization = `Bearer ${key}`;
   } else {
-    requestURL += `?api_key=${key}`;
+    requestURL.searchParams.append('api_key', key);
+  }
+
+  if (appendToResponse) {
+    requestURL.searchParams.append('append_to_response', appendToResponse);
   }
 
   return fetch(requestURL, {
@@ -29,7 +33,7 @@ export async function makeTMDBRequest(url: string): Promise<Response> {
 }
 
 export async function getMovieMediaDetails(id: string): Promise<MovieMedia> {
-  const response = await makeTMDBRequest(`https://api.themoviedb.org/3/movie/${id}`);
+  const response = await makeTMDBRequest(`https://api.themoviedb.org/3/movie/${id}`, 'external_ids');
   const movie = await response.json();
 
   if (movie.success === false) {
@@ -45,13 +49,14 @@ export async function getMovieMediaDetails(id: string): Promise<MovieMedia> {
     title: movie.title,
     releaseYear: Number(movie.release_date.split('-')[0]),
     tmdbId: id,
+    imdbId: movie.imdb_id,
   };
 }
 
 export async function getShowMediaDetails(id: string, seasonNumber: string, episodeNumber: string): Promise<ShowMedia> {
   // * TV shows require the TMDB ID for the series, season, and episode
   // * and the name of the series. Needs multiple requests
-  let response = await makeTMDBRequest(`https://api.themoviedb.org/3/tv/${id}`);
+  let response = await makeTMDBRequest(`https://api.themoviedb.org/3/tv/${id}`, 'external_ids');
   const series = await response.json();
 
   if (series.success === false) {
@@ -91,5 +96,6 @@ export async function getShowMediaDetails(id: string, seasonNumber: string, epis
       number: season.season_number,
       tmdbId: season.id,
     },
+    imdbId: series.external_ids.imdb_id,
   };
 }

src/entrypoint/utils/targets.ts

@@ -5,6 +5,10 @@ export const flags = {
   // the stream is locked on IP, so only works if
   // request maker is same as player (not compatible with proxies)
   IP_LOCKED: 'ip-locked',
+
+  // The source/embed is blocking cloudflare ip's
+  // This flag is not compatible with a proxy hosted on cloudflare
+  CF_BLOCKED: 'cf-blocked',
 } as const;
 
 export type Flags = (typeof flags)[keyof typeof flags];

src/fetchers/simpleProxy.ts

@@ -7,6 +7,8 @@ const headerMap: Record<string, string> = {
   cookie: 'X-Cookie',
   referer: 'X-Referer',
   origin: 'X-Origin',
+  'user-agent': 'X-User-Agent',
+  'x-real-ip': 'X-X-Real-Ip',
 };
 
 const responseHeaderMap: Record<string, string> = {

src/providers/all.ts

@@ -18,11 +18,14 @@ import { vidsrcScraper } from '@/providers/sources/vidsrc/index';
 import { zoechipScraper } from '@/providers/sources/zoechip';
 
 import { closeLoadScraper } from './embeds/closeload';
+import { fileMoonScraper } from './embeds/filemoon';
 import { ridooScraper } from './embeds/ridoo';
 import { smashyStreamDScraper } from './embeds/smashystream/dued';
 import { smashyStreamFScraper } from './embeds/smashystream/video1';
+import { vidplayScraper } from './embeds/vidplay';
 import { ridooMoviesScraper } from './sources/ridomovies';
 import { smashyStreamScraper } from './sources/smashystream';
+import { vidSrcToScraper } from './sources/vidsrcto';
 
 export function gatherAllSources(): Array<Sourcerer> {
   // all sources are gathered here
@@ -37,6 +40,7 @@ export function gatherAllSources(): Array<Sourcerer> {
     lookmovieScraper,
     smashyStreamScraper,
     ridooMoviesScraper,
+    vidSrcToScraper,
   ];
 }
 
@@ -56,5 +60,7 @@ export function gatherAllEmbeds(): Array<Embed> {
     smashyStreamDScraper,
     ridooScraper,
     closeLoadScraper,
+    fileMoonScraper,
+    vidplayScraper,
   ];
 }

src/providers/embeds/filemoon/index.ts

@@ -0,0 +1,58 @@
+import { unpack } from 'unpacker';
+
+import { SubtitleResult } from './types';
+import { makeEmbed } from '../../base';
+import { Caption, getCaptionTypeFromUrl, labelToLanguageCode } from '../../captions';
+
+const evalCodeRegex = /eval\((.*)\)/g;
+const fileRegex = /file:"(.*?)"/g;
+
+export const fileMoonScraper = makeEmbed({
+  id: 'filemoon',
+  name: 'Filemoon',
+  rank: 400,
+  scrape: async (ctx) => {
+    const embedRes = await ctx.proxiedFetcher<string>(ctx.url, {
+      headers: {
+        referer: ctx.url,
+      },
+    });
+    const evalCode = evalCodeRegex.exec(embedRes);
+    if (!evalCode) throw new Error('Failed to find eval code');
+    const unpacked = unpack(evalCode[1]);
+    const file = fileRegex.exec(unpacked);
+    if (!file?.[1]) throw new Error('Failed to find file');
+
+    const url = new URL(ctx.url);
+    const subtitlesLink = url.searchParams.get('sub.info');
+    const captions: Caption[] = [];
+    if (subtitlesLink) {
+      const captionsResult = await ctx.proxiedFetcher<SubtitleResult>(subtitlesLink);
+
+      for (const caption of captionsResult) {
+        const language = labelToLanguageCode(caption.label);
+        const captionType = getCaptionTypeFromUrl(caption.file);
+        if (!language || !captionType) continue;
+        captions.push({
+          id: caption.file,
+          url: caption.file,
+          type: captionType,
+          language,
+          hasCorsRestrictions: false,
+        });
+      }
+    }
+
+    return {
+      stream: [
+        {
+          id: 'primary',
+          type: 'hls',
+          playlist: file[1],
+          flags: [],
+          captions,
+        },
+      ],
+    };
+  },
+});

src/providers/embeds/filemoon/types.ts

@@ -0,0 +1,5 @@
+export type SubtitleResult = {
+  file: string;
+  label: string;
+  kind: string;
+}[];

src/providers/embeds/upcloud.ts

@@ -4,6 +4,9 @@ import { flags } from '@/entrypoint/utils/targets';
 import { makeEmbed } from '@/providers/base';
 import { Caption, getCaptionTypeFromUrl, labelToLanguageCode } from '@/providers/captions';
 
+const origin = 'https://rabbitstream.net';
+const referer = 'https://rabbitstream.net/';
+
 const { AES, enc } = crypto;
 
 interface StreamRes {
@@ -126,6 +129,10 @@ export const upcloudScraper = makeEmbed({
           playlist: sources.file,
           flags: [flags.CORS_ALLOWED],
           captions,
+          preferredHeaders: {
+            Referer: referer,
+            Origin: origin,
+          },
         },
       ],
     };

src/providers/embeds/vidplay/common.ts

@@ -0,0 +1,53 @@
+import { makeFullUrl } from '@/fetchers/common';
+import { decodeData } from '@/providers/sources/vidsrcto/common';
+import { EmbedScrapeContext } from '@/utils/context';
+
+export const vidplayBase = 'https://vidplay.online';
+export const referer = `${vidplayBase}/`;
+
+// This file is based on https://github.com/Ciarands/vidsrc-to-resolver/blob/dffa45e726a4b944cb9af0c9e7630476c93c0213/vidsrc.py#L16
+// Full credits to @Ciarands!
+
+export const getDecryptionKeys = async (ctx: EmbedScrapeContext): Promise<string[]> => {
+  const res = await ctx.fetcher<string>('https://raw.githubusercontent.com/Ciarands/vidsrc-keys/main/keys.json');
+  return JSON.parse(res);
+};
+
+export const getEncodedId = async (ctx: EmbedScrapeContext) => {
+  const url = new URL(ctx.url);
+  const id = url.pathname.replace('/e/', '');
+  const keyList = await getDecryptionKeys(ctx);
+
+  const decodedId = decodeData(keyList[0], id);
+  const encodedResult = decodeData(keyList[1], decodedId);
+  const b64encoded = btoa(encodedResult);
+  return b64encoded.replace('/', '_');
+};
+
+export const getFuTokenKey = async (ctx: EmbedScrapeContext) => {
+  const id = await getEncodedId(ctx);
+  const fuTokenRes = await ctx.proxiedFetcher<string>('/futoken', {
+    baseUrl: vidplayBase,
+    headers: {
+      referer: ctx.url,
+    },
+  });
+  const fuKey = fuTokenRes.match(/var\s+k\s*=\s*'([^']+)'/)?.[1];
+  if (!fuKey) throw new Error('No fuKey found');
+  const tokens = [];
+  for (let i = 0; i < id.length; i += 1) {
+    tokens.push(fuKey.charCodeAt(i % fuKey.length) + id.charCodeAt(i));
+  }
+  return `${fuKey},${tokens.join(',')}`;
+};
+
+export const getFileUrl = async (ctx: EmbedScrapeContext) => {
+  const fuToken = await getFuTokenKey(ctx);
+  return makeFullUrl(`/mediainfo/${fuToken}`, {
+    baseUrl: vidplayBase,
+    query: {
+      ...Object.fromEntries(new URL(ctx.url).searchParams.entries()),
+      autostart: 'true',
+    },
+  });
+};

src/providers/embeds/vidplay/index.ts

@@ -0,0 +1,57 @@
+import { makeEmbed } from '@/providers/base';
+import { Caption, getCaptionTypeFromUrl, labelToLanguageCode } from '@/providers/captions';
+
+import { getFileUrl, referer } from './common';
+import { SubtitleResult, VidplaySourceResponse } from './types';
+
+export const vidplayScraper = makeEmbed({
+  id: 'vidplay',
+  name: 'VidPlay',
+  rank: 401,
+  scrape: async (ctx) => {
+    const fileUrl = await getFileUrl(ctx);
+    const fileUrlRes = await ctx.proxiedFetcher<VidplaySourceResponse>(fileUrl, {
+      headers: {
+        referer: ctx.url,
+      },
+    });
+    if (typeof fileUrlRes.result === 'number') throw new Error('File not found');
+    const source = fileUrlRes.result.sources[0].file;
+
+    const url = new URL(ctx.url);
+    const subtitlesLink = url.searchParams.get('sub.info');
+    const captions: Caption[] = [];
+    if (subtitlesLink) {
+      const captionsResult = await ctx.proxiedFetcher<SubtitleResult>(subtitlesLink);
+
+      for (const caption of captionsResult) {
+        const language = labelToLanguageCode(caption.label);
+        const captionType = getCaptionTypeFromUrl(caption.file);
+        if (!language || !captionType) continue;
+        captions.push({
+          id: caption.file,
+          url: caption.file,
+          type: captionType,
+          language,
+          hasCorsRestrictions: false,
+        });
+      }
+    }
+
+    return {
+      stream: [
+        {
+          id: 'primary',
+          type: 'hls',
+          playlist: source,
+          flags: [],
+          captions,
+          preferredHeaders: {
+            Referer: referer,
+            Origin: referer,
+          },
+        },
+      ],
+    };
+  },
+});

src/providers/embeds/vidplay/types.ts

@@ -0,0 +1,19 @@
+export type VidplaySourceResponse = {
+  result:
+    | {
+        sources: {
+          file: string;
+          tracks: {
+            file: string;
+            kind: string;
+          }[];
+        }[];
+      }
+    | number;
+};
+
+export type SubtitleResult = {
+  file: string;
+  label: string;
+  kind: string;
+}[];

src/providers/embeds/vidsrc.ts

@@ -4,6 +4,14 @@ import { makeEmbed } from '@/providers/base';
 const hlsURLRegex = /file:"(.*?)"/;
 const setPassRegex = /var pass_path = "(.*set_pass\.php.*)";/;
 
+function formatHlsB64(data: string): string {
+  const encodedB64 = data.replace(/\/@#@\/[^=/]+==/g, '');
+  if (encodedB64.match(/\/@#@\/[^=/]+==/)) {
+    return formatHlsB64(encodedB64);
+  }
+  return encodedB64;
+}
+
 export const vidsrcembedScraper = makeEmbed({
   id: 'vidsrcembed', // VidSrc is both a source and an embed host
   name: 'VidSrc',
@@ -15,13 +23,12 @@ export const vidsrcembedScraper = makeEmbed({
       },
     });
 
-    const match = html
+    // When this eventually breaks see the player js @ pjs_main.js
-      .match(hlsURLRegex)?.[1]
+    // If you know what youre doing and are slightly confused about how to reverse this feel free to reach out to ciaran_ds on discord with any queries
-      ?.replace(/(\/\/\S+?=)/g, '')
+    let hlsMatch = html.match(hlsURLRegex)?.[1]?.slice(2);
-      .replace('#2', '');
+    if (!hlsMatch) throw new Error('Unable to find HLS playlist');
-    if (!match) throw new Error('Unable to find HLS playlist');
+    hlsMatch = formatHlsB64(hlsMatch);
-    const finalUrl = atob(match);
+    const finalUrl = atob(hlsMatch);
-
     if (!finalUrl.includes('.m3u8')) throw new Error('Unable to find HLS playlist');
 
     let setPassLink = html.match(setPassRegex)?.[1];

src/providers/sources/lookmovie/index.ts

@@ -32,7 +32,7 @@ async function universalScraper(ctx: MovieScrapeContext | ShowScrapeContext): Pr
 export const lookmovieScraper = makeSourcerer({
   id: 'lookmovie',
   name: 'LookMovie',
-  rank: 1,
+  rank: 700,
   flags: [flags.IP_LOCKED],
   scrapeShow: universalScraper,
   scrapeMovie: universalScraper,

src/providers/sources/remotestream.ts

@@ -4,6 +4,9 @@ import { NotFoundError } from '@/utils/errors';
 
 const remotestreamBase = atob('aHR0cHM6Ly9mc2IuOG1ldDNkdGpmcmNxY2hjb25xcGtsd3hzeGIyb2N1bWMuc3RyZWFt');
 
+const origin = 'https://remotestre.am';
+const referer = 'https://remotestre.am/';
+
 export const remotestreamScraper = makeSourcerer({
   id: 'remotestream',
   name: 'Remote Stream',
@@ -16,9 +19,12 @@ export const remotestreamScraper = makeSourcerer({
     const playlistLink = `${remotestreamBase}/Shows/${ctx.media.tmdbId}/${seasonNumber}/${episodeNumber}/${episodeNumber}.m3u8`;
 
     ctx.progress(30);
-    const streamRes = await ctx.fetcher.full(playlistLink, {
+    const streamRes = await ctx.proxiedFetcher.full(playlistLink, {
-      method: 'HEAD',
+      method: 'GET',
       readHeaders: ['content-type'],
+      headers: {
+        Referer: referer,
+      },
     });
     if (!streamRes.headers.get('content-type')?.toLowerCase().includes('application/x-mpegurl'))
       throw new NotFoundError('No watchable item found');
@@ -33,6 +39,10 @@ export const remotestreamScraper = makeSourcerer({
           playlist: playlistLink,
           type: 'hls',
           flags: [flags.CORS_ALLOWED],
+          preferredHeaders: {
+            Referer: referer,
+            Origin: origin,
+          },
         },
       ],
     };
@@ -41,9 +51,12 @@ export const remotestreamScraper = makeSourcerer({
     const playlistLink = `${remotestreamBase}/Movies/${ctx.media.tmdbId}/${ctx.media.tmdbId}.m3u8`;
 
     ctx.progress(30);
-    const streamRes = await ctx.fetcher.full(playlistLink, {
+    const streamRes = await ctx.proxiedFetcher.full(playlistLink, {
-      method: 'HEAD',
+      method: 'GET',
       readHeaders: ['content-type'],
+      headers: {
+        Referer: referer,
+      },
     });
     if (!streamRes.headers.get('content-type')?.toLowerCase().includes('application/x-mpegurl'))
       throw new NotFoundError('No watchable item found');
@@ -58,6 +71,10 @@ export const remotestreamScraper = makeSourcerer({
           playlist: playlistLink,
           type: 'hls',
           flags: [flags.CORS_ALLOWED],
+          preferredHeaders: {
+            Referer: referer,
+            Origin: origin,
+          },
         },
       ],
     };

src/providers/sources/showbox/index.ts

@@ -42,7 +42,7 @@ export const showboxScraper = makeSourcerer({
   id: 'showbox',
   name: 'Showbox',
   rank: 300,
-  flags: [flags.CORS_ALLOWED],
+  flags: [flags.CORS_ALLOWED, flags.CF_BLOCKED],
   scrapeShow: comboScraper,
   scrapeMovie: comboScraper,
 });

src/providers/sources/showbox/sendRequest.ts

@@ -49,9 +49,9 @@ export const sendRequest = async (ctx: ScrapeContext, data: object, altApi = fal
     headers: {
       Platform: 'android',
       'Content-Type': 'application/x-www-form-urlencoded',
+      'User-Agent': 'okhttp/3.2.0',
     },
     body: formatted,
   });
-
   return JSON.parse(response);
 };

src/providers/sources/vidsrcto/common.ts

@@ -0,0 +1,49 @@
+// This file is based on https://github.com/Ciarands/vidsrc-to-resolver/blob/dffa45e726a4b944cb9af0c9e7630476c93c0213/vidsrc.py#L16
+// Full credits to @Ciarands!
+
+const DECRYPTION_KEY = '8z5Ag5wgagfsOuhz';
+
+export const decodeBase64UrlSafe = (str: string) => {
+  const standardizedInput = str.replace(/_/g, '/').replace(/-/g, '+');
+  const decodedData = atob(standardizedInput);
+
+  const bytes = new Uint8Array(decodedData.length);
+  for (let i = 0; i < bytes.length; i += 1) {
+    bytes[i] = decodedData.charCodeAt(i);
+  }
+
+  return bytes;
+};
+
+export const decodeData = (key: string, data: any) => {
+  const state = Array.from(Array(256).keys());
+  let index1 = 0;
+  for (let i = 0; i < 256; i += 1) {
+    index1 = (index1 + state[i] + key.charCodeAt(i % key.length)) % 256;
+    const temp = state[i];
+    state[i] = state[index1];
+    state[index1] = temp;
+  }
+  index1 = 0;
+  let index2 = 0;
+  let finalKey = '';
+  for (let char = 0; char < data.length; char += 1) {
+    index1 = (index1 + 1) % 256;
+    index2 = (index2 + state[index1]) % 256;
+    const temp = state[index1];
+    state[index1] = state[index2];
+    state[index2] = temp;
+    if (typeof data[char] === 'string') {
+      finalKey += String.fromCharCode(data[char].charCodeAt(0) ^ state[(state[index1] + state[index2]) % 256]);
+    } else if (typeof data[char] === 'number') {
+      finalKey += String.fromCharCode(data[char] ^ state[(state[index1] + state[index2]) % 256]);
+    }
+  }
+  return finalKey;
+};
+
+export const decryptSourceUrl = (sourceUrl: string) => {
+  const encoded = decodeBase64UrlSafe(sourceUrl);
+  const decoded = decodeData(DECRYPTION_KEY, encoded);
+  return decodeURIComponent(decodeURIComponent(decoded));
+};

src/providers/sources/vidsrcto/index.ts

@@ -0,0 +1,84 @@
+import { load } from 'cheerio';
+
+import { SourcererEmbed, SourcererOutput, makeSourcerer } from '@/providers/base';
+import { MovieScrapeContext, ShowScrapeContext } from '@/utils/context';
+
+import { decryptSourceUrl } from './common';
+import { SourceResult, SourcesResult } from './types';
+
+const vidSrcToBase = 'https://vidsrc.to';
+const referer = `${vidSrcToBase}/`;
+
+const universalScraper = async (ctx: ShowScrapeContext | MovieScrapeContext): Promise<SourcererOutput> => {
+  const imdbId = ctx.media.imdbId;
+  const url =
+    ctx.media.type === 'movie'
+      ? `/embed/movie/${imdbId}`
+      : `/embed/tv/${imdbId}/${ctx.media.season.number}/${ctx.media.episode.number}`;
+  const mainPage = await ctx.proxiedFetcher<string>(url, {
+    baseUrl: vidSrcToBase,
+    headers: {
+      referer,
+    },
+  });
+  const mainPage$ = load(mainPage);
+  const dataId = mainPage$('a[data-id]').attr('data-id');
+  if (!dataId) throw new Error('No data-id found');
+  const sources = await ctx.proxiedFetcher<SourcesResult>(`/ajax/embed/episode/${dataId}/sources`, {
+    baseUrl: vidSrcToBase,
+    headers: {
+      referer,
+    },
+  });
+  if (sources.status !== 200) throw new Error('No sources found');
+
+  const embeds: SourcererEmbed[] = [];
+  const embedUrls = [];
+  for (const source of sources.result) {
+    const sourceRes = await ctx.proxiedFetcher<SourceResult>(`/ajax/embed/source/${source.id}`, {
+      baseUrl: vidSrcToBase,
+      headers: {
+        referer,
+      },
+    });
+    const decryptedUrl = decryptSourceUrl(sourceRes.result.url);
+    embedUrls.push(decryptedUrl);
+  }
+
+  // Originally Filemoon does not have subtitles. But we can use the ones from Vidplay.
+  const subtitleUrl = new URL(embedUrls.find((v) => v.includes('sub.info')) ?? '').searchParams.get('sub.info');
+  for (const source of sources.result) {
+    if (source.title === 'Vidplay') {
+      const embedUrl = embedUrls.find((v) => v.includes('vidplay'));
+      if (!embedUrl) continue;
+      embeds.push({
+        embedId: 'vidplay',
+        url: embedUrl,
+      });
+    }
+
+    if (source.title === 'Filemoon') {
+      const embedUrl = embedUrls.find((v) => v.includes('filemoon'));
+      if (!embedUrl) continue;
+      const fullUrl = new URL(embedUrl);
+      if (subtitleUrl) fullUrl.searchParams.set('sub.info', subtitleUrl);
+      embeds.push({
+        embedId: 'filemoon',
+        url: fullUrl.toString(),
+      });
+    }
+  }
+
+  return {
+    embeds,
+  };
+};
+
+export const vidSrcToScraper = makeSourcerer({
+  id: 'vidsrcto',
+  name: 'VidSrcTo',
+  scrapeMovie: universalScraper,
+  scrapeShow: universalScraper,
+  flags: [],
+  rank: 400,
+});

src/providers/sources/vidsrcto/types.ts

@@ -0,0 +1,15 @@
+export type VidSrcToResponse<T> = {
+  status: number;
+  result: T;
+};
+
+export type SourcesResult = VidSrcToResponse<
+  {
+    id: string;
+    title: 'Filemoon' | 'Vidplay';
+  }[]
+>;
+
+export type SourceResult = VidSrcToResponse<{
+  url: string;
+}>;

src/runners/individualRunner.ts

@@ -58,6 +58,14 @@ export async function scrapeInvidualSource(
   }
 
   if (!output) throw new Error('output is null');
+
+  // filter output with only valid embeds that are not disabled
+  output.embeds = output.embeds.filter((embed) => {
+    const e = list.embeds.find((v) => v.id === embed.embedId);
+    if (!e || e.disabled) return false;
+    return true;
+  });
+
   if ((!output.stream || output.stream.length === 0) && output.embeds.length === 0)
     throw new NotFoundError('No streams found');
   return output;

src/runners/runner.ts

@@ -116,8 +116,14 @@ export async function runAllProviders(list: ProviderList, ops: ProviderRunnerOpt
       };
     }
 
-    // run embed scrapers on listed embeds
+    // filter disabled and run embed scrapers on listed embeds
-    const sortedEmbeds = output.embeds.sort((a, b) => embedIds.indexOf(a.embedId) - embedIds.indexOf(b.embedId));
+    const sortedEmbeds = output.embeds
+      .filter((embed) => {
+        const e = list.embeds.find((v) => v.id === embed.embedId);
+        if (!e || e.disabled) return false;
+        return true;
+      })
+      .sort((a, b) => embedIds.indexOf(a.embedId) - embedIds.indexOf(b.embedId));
 
     if (sortedEmbeds.length > 0) {
       ops.events?.discoverEmbeds?.({

vite.config.js

@@ -5,7 +5,9 @@ const dts = require('vite-plugin-dts');
 const pkg = require('./package.json');
 const fs = require('fs/promises');
 
-const main = path.resolve(__dirname, 'src/index.ts');
+const shouldTestProviders = process.env.MW_TEST_PROVIDERS === "true"
+let tests = ['src/__test__/standard/**/*.test.ts'];
+if (shouldTestProviders) tests = ['src/__test__/providers/**/*.test.ts']
 
 module.exports = defineConfig({
   plugins: [
@@ -34,10 +36,13 @@ module.exports = defineConfig({
     outDir: 'lib',
 
     lib: {
-      entry: main,
+      entry: path.resolve(__dirname, 'src/index.ts'),
       name: 'index',
       fileName: 'index',
       formats: ['umd', 'es'],
     },
   },
+  test: {
+    include: tests
+  }
 });