Pool map and external collections

 There are two types of cues' folders: Scripthea own and External collections with downloadable more than 1.5 million of unique prompts.   

Pool Map (Scripthea own cues and external collections)

Cues pool is a group of cue lists. Currently there are only two of them (A & B).

Click on the left of the cue item to check/uncheck to enable/disable it. Click on the right to select the cue item for transfer into the other pool.

From Pool Map tab you can transfer cue lists from one pool to the other (left/right arrow) or hide a cue list if needed (checkboxes). You can reorder using up/down arrow, first select an item and then move it. In similar manner operate rename (pencil) button and delete (cross) button. The third button from the bottom in case of ext. collection will copy the extraction options from the selected cues list to the extraction panel.

Any new (unseen before) cues list will appear in Pool A.

Another way to group cues is to select (and load) different cues' folder. All the cues' folders must be subdirectories of the <root> cues folder (Cues), fixed in Scripthea folder. When you open Pool Map tab Scripthea will read all cues' folders and list them in the combobox. Then you select one of them and load it. Each of the loaded cues' folders has its own pool map. 

After installation of Scripthea will have two types of cues' lists: Scripthea own cues' lists and External collections prompts. The way to use ext. collections  as regular cue list is to extract a cue list. You will see the different color items in combo-box for each kind of folder.

Once you load an external collection you need to extract some cues in cues' lists. Available extraction options are applied to external collection one after another from top to bottom:

  • segment - consider only part of the collection;
  • size in words - limit the prompt length;
  • for some external collections you can add a categories selection and sensitivity to the selection criteria (see on the right). All the prompts in such a collection are categorized in up to 3 categories, using by Meta's LLama 3.3 70B model in LM Studio. The categories attributed to one prompt have weights in percents. The sensitivity is the threshold above which any of selected above categories will trigger selection. A sensible sensitivity would be between 20% and 70%. Very useful feature, try it for more clarity.
  • compare the text from Text Matching to extracted list at that point using one of three methods:
  • Simple: look for a particular word(s) in a Text Matching field, that is simple search (as in the viewer)
  • RegEx. (Regular Expressions) is very powerful search tool but it uses specific syntax for matching e.g. (girl|woman) in RegEx mode will extract all the prompts containing the words "girl" or "woman". Here are nice quick reference and tutorial or you can browse around for tutorials.
  • Semantic: this method ranks all the cues at this stage against the text by semantic similarity (cosine normalized distances) and extract the best (top) ones to pass them further. If the text field is empty or Top number is 0 Scripthea will skip semantic matching. You can interrupt the matching from the red button next to Top number.
    The semantic model is CLIP derivate (originally by OpenAI) and library used is machine learning ONNX of Microsoft. The latter is installed along the all other dll's but CLIP model is too large (>250 Mb) to be treated the same way. If you decide to use semantic matching method you need to download the model which is part of Semantic Extension (see below).
    The semantic method is noticeable slower than the others but because it runs on a parallel thread you can do something else inside Scripthea while waiting for the semantic matching to finish. You will be reminded for the ongoing semantic matching by "waiting" shape of the mouse cursor. 
  • limit the length of result list by random sampling to a certain cue count. 

All the extraction options are checkable, and applied in order from top to bottom. Nevertheless the final cues count cannot exceed 1000 cues, as recommended count would be below 500.

Scripthea proper cues are well groomed, still, the total number so far is less than 1000. While the external collections count is in millions.  If you need to scan over cues from different cues folders, load them one by one and use Append to the preview button.

External collections manager

Using external collections will give you almost unlimited (>1.5 million so far) number of unique prompts.

All the collections come from dataset section of HuggingFace website with filter "prompts". In each collection there is a link to the Data card of that particular dataset. The primary selected datasets are converted to simple text file (.STX) with each prompts is a row OR json lines file (.SJL) where one row is json coded prompts and some meta information for that prompt.

Then prompts are filtered using variety of criteria like words validity, size (in words) range, removing porn, etc. Each new prompt is tested against all the prompts already in the collections pot for duplication so all prompts are unique not only within a particular collection but within all the collections.

Some of collections are evaluated for quality by LLM, others (like k-detailed) are classified as belonging (in percentages) to up to three out of 8 categories:

1. Fine Art, Classical & Historical Art
2. Impressionism, Post-impressionism & Expressionism
3. Abstract, Post-modern & Psychedelic
4. Surrealism & Dadaism
5. Futuristic, Mythical & Historical Fantasy
6. Popular culture, Current trends & celebrities
7. Anime, Cartoon & Illustration
8. Realistic & Photographic

When you enter Pool Map tab of cues panel, on the top right corner you will round button with wavy pattern, that is the entry point to External Collection Manager. There you will see a list of all available collections located locally or remotely. You can download and install any collection you don't have locally (or maybe there is a new version). On the top right you can do that with one button click.

Sometimes, mostly due to poor connection the download may fail. In this case (the button down left) you can opt to downloading the zip file of the collection using your preferred browser which probably has more robust way to download files. After you download the zip you can can click on button on the right to install that collection locally.

Semantic Extension

You can download and install Semantic Extension. Similar to external collection manager you can use one button action (Download, unzip and install) or if you face some slow internet issues, step by step (two buttons action). At the end of the extension installation a validation of integrity is performed and corresponding message will let you know the result.