WIP - add extractor, generate snippet_data

This commit is contained in:
Stefan Fejes
2019-08-20 15:52:05 +02:00
parent 88084d3d30
commit cc8f1d8a7a
37396 changed files with 4588842 additions and 133 deletions

192
node_modules/jest-worker/README.md generated vendored Normal file
View File

@ -0,0 +1,192 @@
# jest-worker
Module for executing heavy tasks under forked processes in parallel, by providing a `Promise` based interface, minimum overhead, and bound workers.
The module works by providing an absolute path of the module to be loaded in all forked processes. Files relative to a node module are also accepted. All methods are exposed on the parent process as promises, so they can be `await`'ed. Child (worker) methods can either be synchronous or asynchronous.
The module also implements support for bound workers. Binding a worker means that, based on certain parameters, the same task will always be executed by the same worker. The way bound workers work is by using the returned string of the `computeWorkerKey` method. If the string was used before for a task, the call will be queued to the related worker that processed the task earlier; if not, it will be executed by the first available worker, then sticked to the worker that executed it; so the next time it will be processed by the same worker. If you have no preference on the worker executing the task, but you have defined a `computeWorkerKey` method because you want _some_ of the tasks to be sticked, you can return `null` from it.
The list of exposed methods can be explicitly provided via the `exposedMethods` option. If it is not provided, it will be obtained by requiring the child module into the main process, and analyzed via reflection. Check the "minimal example" section for a valid one.
## Install
```sh
$ yarn add jest-worker
```
## Example
This example covers the minimal usage:
### File `parent.js`
```javascript
import Worker from 'jest-worker';
async function main() {
const worker = new Worker(require.resolve('./worker'));
const result = await worker.hello('Alice'); // "Hello, Alice"
}
main();
```
### File `worker.js`
```javascript
export function hello(param) {
return 'Hello, ' + param;
}
```
## API
The only exposed method is a constructor (`Worker`) that is initialized by passing the worker path, plus an options object.
### `workerPath: string` (required)
Node module name or absolute path of the file to be loaded in the child processes. Use `require.resolve` to transform a relative path into an absolute one.
### `options: Object` (optional)
#### `exposedMethods: $ReadOnlyArray<string>` (optional)
List of method names that can be called on the child processes from the parent process. You cannot expose any method named like a public `Worker` method, or starting with `_`. If you use method auto-discovery, then these methods will not be exposed, even if they exist.
#### `numWorkers: number` (optional)
Amount of workers to spawn. Defaults to the number of CPUs minus 1.
#### `maxRetries: number` (optional)
Maximum amount of times that a dead child can be re-spawned, per call. Defaults to `3`, pass `Infinity` to allow endless retries.
#### `forkOptions: Object` (optional)
Allow customizing all options passed to `childProcess.fork`. By default, some values are set (`cwd`, `env` and `execArgv`), but you can override them and customize the rest. For a list of valid values, check [the Node documentation](https://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options).
#### `computeWorkerKey: (method: string, ...args: Array<any>) => ?string` (optional)
Every time a method exposed via the API is called, `computeWorkerKey` is also called in order to bound the call to a worker. This is useful for workers that are able to cache the result or part of it. You bound calls to a worker by making `computeWorkerKey` return the same identifier for all different calls. If you do not want to bind the call to any worker, return `null`.
The callback you provide is called with the method name, plus all the rest of the arguments of the call. Thus, you have full control to decide what to return. Check a practical example on bound workers under the "bound worker usage" section.
By default, no process is bound to any worker.
## Worker
The returned `Worker` instance has all the exposed methods, plus some additional ones to interact with the workers itself:
### `getStdout(): Readable`
Returns a `ReadableStream` where the standard output of all workers is piped. Note that the `silent` option of the child workers must be set to `true` to make it work. This is the default set by `jest-worker`, but keep it in mind when overriding options through `forkOptions`.
### `getStderr(): Readable`
Returns a `ReadableStream` where the standard error of all workers is piped. Note that the `silent` option of the child workers must be set to `true` to make it work. This is the default set by `jest-worker`, but keep it in mind when overriding options through `forkOptions`.
### `end()`
Finishes the workers by killing all workers. No further calls can be done to the `Worker` instance.
**Note:** Each worker has a unique id (index that starts with `1`) which is available on `process.env.JEST_WORKER_ID`
# More examples
## Standard usage
This example covers the standard usage:
### File `parent.js`
```javascript
import Worker from 'jest-worker';
async function main() {
const myWorker = new Worker(require.resolve('./worker'), {
exposedMethods: ['foo', 'bar', 'getWorkerId'],
numWorkers: 4,
});
console.log(await myWorker.foo('Alice')); // "Hello from foo: Alice"
console.log(await myWorker.bar('Bob')); // "Hello from bar: Bob"
console.log(await myWorker.getWorkerId()); // "3" -> this message has sent from the 3rd worker
myWorker.end();
}
main();
```
### File `worker.js`
```javascript
export function foo(param) {
return 'Hello from foo: ' + param;
}
export function bar(param) {
return 'Hello from bar: ' + param;
}
export function getWorkerId() {
return process.env.JEST_WORKER_ID;
}
```
## Bound worker usage:
This example covers the usage with a `computeWorkerKey` method:
### File `parent.js`
```javascript
import Worker from 'jest-worker';
async function main() {
const myWorker = new Worker(require.resolve('./worker'), {
computeWorkerKey: (method, filename) => filename,
});
// Transform the given file, within the first available worker.
console.log(await myWorker.transform('/tmp/foo.js'));
// Wait a bit.
await sleep(10000);
// Transform the same file again. Will immediately return because the
// transformed file is cached in the worker, and `computeWorkerKey` ensures
// the same worker that processed the file the first time will process it now.
console.log(await myWorker.transform('/tmp/foo.js'));
myWorker.end();
}
main();
```
### File `worker.js`
```javascript
import babel from 'babel-core';
const cache = Object.create(null);
export function transform(filename) {
if (cache[filename]) {
return cache[filename];
}
// jest-worker can handle both immediate results and thenables. If a
// thenable is returned, it will be await'ed until it resolves.
return new Promise((resolve, reject) => {
babel.transformFile(filename, (err, result) => {
if (err) {
reject(err);
} else {
resolve((cache[filename] = result));
}
});
});
}
```

102
node_modules/jest-worker/build/child.js generated vendored Normal file
View File

@ -0,0 +1,102 @@
/**
* Copyright (c) 2017-present, Facebook, Inc. All rights reserved.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*
*
*/
'use strict';
var _types;
function _load_types() {
return (_types = require('./types'));
}
let file = null;
/**
* This file is a small bootstrapper for workers. It sets up the communication
* between the worker and the parent process, interpreting parent messages and
* sending results back.
*
* The file loaded will be lazily initialized the first time any of the workers
* is called. This is done for optimal performance: if the farm is initialized,
* but no call is made to it, child Node processes will be consuming the least
* possible amount of memory.
*
* If an invalid message is detected, the child will exit (by throwing) with a
* non-zero exit code.
*/
process.on('message', (request /* Should be ChildMessage */) => {
switch (request[0]) {
case (_types || _load_types()).CHILD_MESSAGE_INITIALIZE:
file = request[2];
break;
case (_types || _load_types()).CHILD_MESSAGE_CALL:
execMethod(request[2], request[3]);
break;
case (_types || _load_types()).CHILD_MESSAGE_END:
process.exit(0);
break;
default:
throw new TypeError(
'Unexpected request from parent process: ' + request[0]
);
}
});
function reportSuccess(result) {
if (!process || !process.send) {
throw new Error('Child can only be used on a forked process');
}
process.send([(_types || _load_types()).PARENT_MESSAGE_OK, result]);
}
function reportError(error) {
if (!process || !process.send) {
throw new Error('Child can only be used on a forked process');
}
if (error == null) {
error = new Error('"null" or "undefined" thrown');
}
process.send([
(_types || _load_types()).PARENT_MESSAGE_ERROR,
error.constructor && error.constructor.name,
error.message,
error.stack,
// $FlowFixMe: this is safe to just inherit from Object.
typeof error === 'object' ? Object.assign({}, error) : error
]);
}
function execMethod(method, args) {
// $FlowFixMe: This has to be a dynamic require.
const main = require(file);
let result;
try {
if (method === 'default') {
result = (main.__esModule ? main['default'] : main).apply(global, args);
} else {
result = main[method].apply(main, args);
}
} catch (err) {
reportError(err);
return;
}
if (result && typeof result.then === 'function') {
result.then(reportSuccess, reportError);
} else {
reportSuccess(result);
}
}

255
node_modules/jest-worker/build/index.js generated vendored Normal file
View File

@ -0,0 +1,255 @@
/**
* Copyright (c) 2017-present, Facebook, Inc. All rights reserved.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*
*
*/
'use strict';
Object.defineProperty(exports, '__esModule', {
value: true
});
var _mergeStream;
function _load_mergeStream() {
return (_mergeStream = _interopRequireDefault(require('merge-stream')));
}
var _os;
function _load_os() {
return (_os = _interopRequireDefault(require('os')));
}
var _path;
function _load_path() {
return (_path = _interopRequireDefault(require('path')));
}
var _types;
function _load_types() {
return (_types = require('./types'));
}
var _worker;
function _load_worker() {
return (_worker = _interopRequireDefault(require('./worker')));
}
function _interopRequireDefault(obj) {
return obj && obj.__esModule ? obj : {default: obj};
}
/* istanbul ignore next */
const emptyMethod = () => {};
/**
* The Jest farm (publicly called "Worker") is a class that allows you to queue
* methods across multiple child processes, in order to parallelize work. This
* is done by providing an absolute path to a module that will be loaded on each
* of the child processes, and bridged to the main process.
*
* Bridged methods are specified by using the "exposedMethods" property of the
* options "object". This is an array of strings, where each of them corresponds
* to the exported name in the loaded module.
*
* You can also control the amount of workers by using the "numWorkers" property
* of the "options" object, and the settings passed to fork the process through
* the "forkOptions" property. The amount of workers defaults to the amount of
* CPUS minus one.
*
* Queueing calls can be done in two ways:
* - Standard method: calls will be redirected to the first available worker,
* so they will get executed as soon as they can.
*
* - Sticky method: if a "computeWorkerKey" method is provided within the
* config, the resulting string of this method will be used as a key.
* Everytime this key is returned, it is guaranteed that your job will be
* processed by the same worker. This is specially useful if your workers are
* caching results.
*/
exports.default = class {
constructor(workerPath) {
let options =
arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : {};
const numWorkers =
options.numWorkers || (_os || _load_os()).default.cpus().length - 1;
const workers = new Array(numWorkers);
const stdout = (0, (_mergeStream || _load_mergeStream()).default)();
const stderr = (0, (_mergeStream || _load_mergeStream()).default)();
if (!(_path || _load_path()).default.isAbsolute(workerPath)) {
workerPath = require.resolve(workerPath);
}
const sharedWorkerOptions = {
forkOptions: options.forkOptions || {},
maxRetries: options.maxRetries || 3,
workerPath
};
for (let i = 0; i < numWorkers; i++) {
const workerOptions = Object.assign({}, sharedWorkerOptions, {
workerId: i + 1
});
const worker = new (_worker || _load_worker()).default(workerOptions);
const workerStdout = worker.getStdout();
const workerStderr = worker.getStderr();
if (workerStdout) {
stdout.add(workerStdout);
}
if (workerStderr) {
stderr.add(workerStderr);
}
workers[i] = worker;
}
let exposedMethods = options.exposedMethods;
// If no methods list is given, try getting it by auto-requiring the module.
if (!exposedMethods) {
// $FlowFixMe: This has to be a dynamic require.
const child = require(workerPath);
exposedMethods = Object.keys(child).filter(
name => typeof child[name] === 'function'
);
if (typeof child === 'function') {
exposedMethods.push('default');
}
}
exposedMethods.forEach(name => {
if (name.startsWith('_')) {
return;
}
if (this.constructor.prototype.hasOwnProperty(name)) {
throw new TypeError('Cannot define a method called ' + name);
}
// $FlowFixMe: dynamic extension of the class instance is expected.
this[name] = this._makeCall.bind(this, name);
});
this._stdout = stdout;
this._stderr = stderr;
this._ending = false;
this._cacheKeys = Object.create(null);
this._options = options;
this._workers = workers;
this._offset = 0;
}
getStdout() {
return this._stdout;
}
getStderr() {
return this._stderr;
}
end() {
if (this._ending) {
throw new Error('Farm is ended, no more calls can be done to it');
}
const workers = this._workers;
// We do not cache the request object here. If so, it would only be only
// processed by one of the workers, and we want them all to close.
for (let i = 0; i < workers.length; i++) {
workers[i].send(
[(_types || _load_types()).CHILD_MESSAGE_END, false],
emptyMethod,
emptyMethod
);
}
this._ending = true;
}
// eslint-disable-next-line no-unclear-flowtypes
_makeCall(method) {
for (
var _len = arguments.length,
args = Array(_len > 1 ? _len - 1 : 0),
_key = 1;
_key < _len;
_key++
) {
args[_key - 1] = arguments[_key];
}
if (this._ending) {
throw new Error('Farm is ended, no more calls can be done to it');
}
return new Promise((resolve, reject) => {
const computeWorkerKey = this._options.computeWorkerKey;
const workers = this._workers;
const length = workers.length;
const cacheKeys = this._cacheKeys;
const request = [
(_types || _load_types()).CHILD_MESSAGE_CALL,
false,
method,
args
];
let worker = null;
let hash = null;
if (computeWorkerKey) {
hash = computeWorkerKey.apply(this, [method].concat(args));
worker = hash == null ? null : cacheKeys[hash];
}
// Do not use a fat arrow since we need the "this" value, which points to
// the worker that executed the call.
const onProcessStart = worker => {
if (hash != null) {
cacheKeys[hash] = worker;
}
};
const onProcessEnd = (error, result) => {
if (error) {
reject(error);
} else {
resolve(result);
}
};
// If a worker is pre-selected, use it...
if (worker) {
worker.send(request, onProcessStart, onProcessEnd);
return;
}
// ... otherwise use all workers, so the first one available will pick it.
for (let i = 0; i < length; i++) {
workers[(i + this._offset) % length].send(
request,
onProcessStart,
onProcessEnd
);
}
this._offset++;
});
}
};

34
node_modules/jest-worker/build/types.js generated vendored Normal file
View File

@ -0,0 +1,34 @@
/**
* Copyright (c) 2017-present, Facebook, Inc. All rights reserved.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*
*
*/
'use strict';
// Because of the dynamic nature of a worker communication process, all messages
// coming from any of the other processes cannot be typed. Thus, many types
// include "any" as a flow type, which is (unfortunately) correct here.
/* eslint-disable no-unclear-flowtypes */
Object.defineProperty(exports, '__esModule', {
value: true
});
const CHILD_MESSAGE_INITIALIZE = (exports.CHILD_MESSAGE_INITIALIZE = 0);
const CHILD_MESSAGE_CALL = (exports.CHILD_MESSAGE_CALL = 1);
const CHILD_MESSAGE_END = (exports.CHILD_MESSAGE_END = 2);
const PARENT_MESSAGE_OK = (exports.PARENT_MESSAGE_OK = 0);
const PARENT_MESSAGE_ERROR = (exports.PARENT_MESSAGE_ERROR = 1);
// Option objects.
// Messages passed from the parent to the children.
// Messages passed from the children to the parent.
// Queue types.

209
node_modules/jest-worker/build/worker.js generated vendored Normal file
View File

@ -0,0 +1,209 @@
/**
* Copyright (c) 2017-present, Facebook, Inc. All rights reserved.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*
*
*/
'use strict';
Object.defineProperty(exports, '__esModule', {
value: true
});
var _child_process;
function _load_child_process() {
return (_child_process = _interopRequireDefault(require('child_process')));
}
var _types;
function _load_types() {
return (_types = require('./types'));
}
function _interopRequireDefault(obj) {
return obj && obj.__esModule ? obj : {default: obj};
}
/**
* This class wraps the child process and provides a nice interface to
* communicate with. It takes care of:
*
* - Re-spawning the process if it dies.
* - Queues calls while the worker is busy.
* - Re-sends the requests if the worker blew up.
*
* The reason for queueing them here (since childProcess.send also has an
* internal queue) is because the worker could be doing asynchronous work, and
* this would lead to the child process to read its receiving buffer and start a
* second call. By queueing calls here, we don't send the next call to the
* children until we receive the result of the previous one.
*
* As soon as a request starts to be processed by a worker, its "processed"
* field is changed to "true", so that other workers which might encounter the
* same call skip it.
*/
exports.default = class {
constructor(options) {
this._options = options;
this._queue = null;
this._initialize();
}
getStdout() {
return this._child.stdout;
}
getStderr() {
return this._child.stderr;
}
send(request, onProcessStart, onProcessEnd) {
const item = {next: null, onProcessEnd, onProcessStart, request};
if (this._last) {
this._last.next = item;
} else {
this._queue = item;
}
this._last = item;
this._process();
}
_initialize() {
const child = (_child_process || _load_child_process()).default.fork(
require.resolve('./child'),
// $FlowFixMe: Flow does not work well with Object.assign.
Object.assign(
{
cwd: process.cwd(),
env: Object.assign({}, process.env, {
JEST_WORKER_ID: this._options.workerId
}),
// Suppress --debug / --inspect flags while preserving others (like --harmony).
execArgv: process.execArgv.filter(v => !/^--(debug|inspect)/.test(v)),
silent: true
},
this._options.forkOptions
)
);
child.on('message', this._receive.bind(this));
child.on('exit', this._exit.bind(this));
// $FlowFixMe: wrong "ChildProcess.send" signature.
child.send([
(_types || _load_types()).CHILD_MESSAGE_INITIALIZE,
false,
this._options.workerPath
]);
this._retries++;
this._child = child;
this._busy = false;
// If we exceeded the amount of retries, we will emulate an error reply
// coming from the child. This avoids code duplication related with cleaning
// the queue, and scheduling the next call.
if (this._retries > this._options.maxRetries) {
const error = new Error('Call retries were exceeded');
this._receive([
(_types || _load_types()).PARENT_MESSAGE_ERROR,
error.name,
error.message,
error.stack,
{type: 'WorkerError'}
]);
}
}
_process() {
if (this._busy) {
return;
}
let item = this._queue;
// Calls in the queue might have already been processed by another worker,
// so we have to skip them.
while (item && item.request[1]) {
item = item.next;
}
this._queue = item;
if (item) {
// Flag the call as processed, so that other workers know that they don't
// have to process it as well.
item.request[1] = true;
// Tell the parent that this item is starting to be processed.
item.onProcessStart(this);
this._retries = 0;
this._busy = true;
// $FlowFixMe: wrong "ChildProcess.send" signature.
this._child.send(item.request);
} else {
this._last = item;
}
}
_receive(response /* Should be ParentMessage */) {
const item = this._queue;
if (!item) {
throw new TypeError('Unexpected response with an empty queue');
}
const onProcessEnd = item.onProcessEnd;
this._busy = false;
this._process();
switch (response[0]) {
case (_types || _load_types()).PARENT_MESSAGE_OK:
onProcessEnd(null, response[1]);
break;
case (_types || _load_types()).PARENT_MESSAGE_ERROR:
let error = response[4];
if (error != null && typeof error === 'object') {
const extra = error;
const NativeCtor = global[response[1]];
const Ctor = typeof NativeCtor === 'function' ? NativeCtor : Error;
error = new Ctor(response[2]);
// $FlowFixMe: adding custom properties to errors.
error.type = response[1];
error.stack = response[3];
for (const key in extra) {
// $FlowFixMe: adding custom properties to errors.
error[key] = extra[key];
}
}
onProcessEnd(error, null);
break;
default:
throw new TypeError('Unexpected response from worker: ' + response[0]);
}
}
_exit(exitCode) {
if (exitCode !== 0) {
this._initialize();
}
}
};

43
node_modules/jest-worker/package.json generated vendored Normal file
View File

@ -0,0 +1,43 @@
{
"_from": "jest-worker@^23.2.0",
"_id": "jest-worker@23.2.0",
"_inBundle": false,
"_integrity": "sha1-+vcGqNo2+uYOsmlXJX+ntdjqArk=",
"_location": "/jest-worker",
"_phantomChildren": {},
"_requested": {
"type": "range",
"registry": true,
"raw": "jest-worker@^23.2.0",
"name": "jest-worker",
"escapedName": "jest-worker",
"rawSpec": "^23.2.0",
"saveSpec": null,
"fetchSpec": "^23.2.0"
},
"_requiredBy": [
"/gatsby"
],
"_resolved": "https://registry.npmjs.org/jest-worker/-/jest-worker-23.2.0.tgz",
"_shasum": "faf706a8da36fae60eb26957257fa7b5d8ea02b9",
"_spec": "jest-worker@^23.2.0",
"_where": "/Users/stefanfejes/Projects/30-seconds-of-python-code/node_modules/gatsby",
"bugs": {
"url": "https://github.com/facebook/jest/issues"
},
"bundleDependencies": false,
"dependencies": {
"merge-stream": "^1.0.1"
},
"deprecated": false,
"description": "Module for executing heavy tasks under forked processes in parallel, by providing a `Promise` based interface, minimum overhead, and bound workers.",
"homepage": "https://github.com/facebook/jest#readme",
"license": "MIT",
"main": "build/index.js",
"name": "jest-worker",
"repository": {
"type": "git",
"url": "git+https://github.com/facebook/jest.git"
},
"version": "23.2.0"
}