XML-Based Application Description Information

TOPIC

This article explores how different components might be used in a future version of Amiga Forever to manage application-specific information and provide a seamless one-click experience for downloading, installing, accessing and launching Amiga applications.

DISCUSSION

This information is not current. The concepts outlined in this page evolved and were implemented in Amiga Forever 2008. Please refer to the RetroPlatform Technical Reference for additional information.

Project Goals

The framework proposed and discussed in this document has the following goals:

• Increased accessibility of Amiga applications (one-click experience for users, rather than browse, download, unzip, install, locate, configure, load, etc., in full respect of hosting site requirements, e.g. bandwidth-saving options, etc.)
• Preservation of Amiga application configuration data together with software image (e.g. the recommended Amiga configuration and actions required to run a specific Amiga game or demo, consisting of one ore more images)
• Cross-system preservation and sharing of additional application data (publisher information, programmers, box and disk scans, screenshots, release notes, etc.) which is not strictly required for accessing or running an application.

We believe that in particular the synergistic combination of the above first two points will, in practice, have surprising effects, igniting a chain of real use, appreciation and support.

Time wise, we have a short-term goal and a longer term goal:

• In the short term, it is considered both desirable and practical to establish a shared XML-based data structure to represent an existing and agreeable common subset of information (i.e. short description and configuration) available on most Amiga game and demo sites for automated use by front-end applications and in emulation software
• In the longer term, we believe that it would be more efficient for all interested sites and organizations to consider adopting similar, if not identical data structures even for data fields which are more aimed at preservation of information of historical interest (e.g. screenshots, scans, detailed information), rather than quick access and playability (i.e. short description and configuration); this involves the definition of a more detailed, discussed and extended data set, which may take longer to define and which is likely to mutate and evolve over time much more than the simple description and configuration data

Indirect effects are expected to include:

• Improved cooperation across organizations to catalog Amiga publisher and software information with less duplication of work
• Creation of new types of rapid-access front-ends
• Creation of tools to sort through thousands of disk images, and to automatically associate them to descriptions and configurations, as well as to detect differences that require human attention (e.g. identically named image files with different CRC32-content or vice versa, different configurations for identical CRC32-content, etc.)
• New ways to face the bandwidth problem: we believe that a free system can greatly benefit from automated techniques and tools to identify, locate and access content across... peers (both on the server side and on the client side)

Although platform-neutral and application-neutral in its intended implementation framework, in practice the effort has a priority focus on the Amiga computer (and its configurations and procedures) and specifically on game and demo applications (and related software/media images).

Simplicity is a main priority in this effort. We believe that if we don't keep things really simple, it will be very difficult, if not impossible, to achieve an initial implementation (especially for the short description and configuration). For this reason the proposed specifications use a "least common denominator" approach that would require some general programming to convert between database formats, but only little or no per-title editing work (to associate a configuration to one or more images which make up an application) to make the data available.

Cloanto itself is to develop and offer a client-side implementation, with the hope that it be helpful to users (not just Amiga Forever users), and of inspiration to other developers.

Terminology

• "Title", or "Application": refers to a specific game or demo or other application (software or data)
• "Advertising": the process of making available a list of multiple title descriptions

Project Components

We consider it both desirable and possible for the Amiga emulation community in general, and for us as Amiga Forever users, to include a set of third-party and in-house components which act in synergy to achieve a completely new user experience.

Specifications developed collectively:

• Format for Application Description and Configuration Data (one XML file per application, e.g. per game or per demo)
• Format for Advertising of Multiple Applications (one XML file contains multiple compact descriptions, e.g. a list of all titles hosted on a site)
• Master list of supported sites (a formal or informal list of sites supporting advertising of multiple applications)

Components developed by third parties:

• New versions of WinUAE and WinFellow, able to load XML-based computer configuration information, software image data and other information required to launch an application
• Websites offering such "configuration-enabled" disk image archives, with the additional option to offer the archives not just as an HTTP download, but also (possibly at a later stage) with a new "auto-install" link

Components developed by Cloanto:

• "MenuList" ActiveX control in Games tab of Amiga Forever launcher (a browser container window built with MenuBox, currently only for Windows), used to monitor Games folder for XML game description files and build and refresh the list of games accordingly (including preview thumbnail on Select action and passing XML data to emulation software on Open action)
• Same as above, for Demos tab and folder
• Software Director MIME type (cross-platform) and logic (initially only for Windows) to allow users to invoke an automatic installation (with security confirmation dialog as has reliably been done for years) of an Amiga game and configuration directly from a website (with a single click, similar to .torrent MIME type and links, actually may include Torrent client functionality)
• Public UUID Generator (optional, any RFC-compliant UUID generator will work)
• Long-term hosting of XML schema files, if desired

On its side, Cloanto intends to make available the Software Director (formerly Software Manager) software with the required publisher entries (probably one for each supported download site) not only as part of Amiga Forever, but also as a free download. Similarly, agreements may be worked out to provide the MenuList control for free inclusion in WinUAE and WinFellow, if it becomes desirable for these applications to directly include a front-end/loader.

Overview of Application Description and Configuration Data

In order to fulfill the possibilities described here, the XML file associated to an application would need to contain at least four parts:

• Compact application description (essential textual information, plus a single reference screenshot thumbnail), for listing and for advertising purposes, processed by launcher/front-end, tentatively does not change so much over time
• Extended application description (with multiple original size screenshots, credits, box scans, disk scans, etc.) for additional uses, not strictly required for "preview" purposes, can easily be extended as the need for new fields emerges
• Application configuration data, for processing by the emulation software
• A UUID (unique identifier), to identify XML files referring to the same application, for handling duplicates, to properly list an application as "Installed" instead of "Available", to match available files with availability on download sites, etc.

Compact Application Description Data Detail

It is envisioned that the Compact Application Description part be used for listing purposes, e.g. in the Games tab of the Amiga Forever launcher, in a possible game/configuration loader inside WinUAE, etc. This part does not serve any configuration purposes, it is for description only.

We would like to propose the use of the following minimal set of Application Description fields for immediate practical use by application launchers (the MenuList component is already complete and ready for testing):

• Data Version (required field)
• UUID (required field)
• Class (required field, defining the platform, in a broad sense, e.g. "amiga")
• Type (required field, e.g. currently one or more of "system", "game", "demo", "application", "other-software", "data", "video", "audio", "other")
• Entity (required field, publishing entity, e.g. game publisher, or demo group, or "?" if unknown)
• Title (required field)
• Version (for separate display, or ready to be appended to Title separated by a single space character, e.g. "1.2.3")
• Variant (e.g. "ocs", "ecs", "aga", "cdtv", "cd32" for Amiga applications, this string is for display purposes, and it refers to the specific title, it is not for configuration purposes)
• SystemROM (e.g. "100", "110", "120", "130", "204", "205", "300", "310", "310-cd32" for publicly released Amiga ROMs, this string is for launcher purposes, and it refers to the specific title, it is not for configuration purposes)
• Year (if used, four digits as per "common era" dating system, or exact string "198?", "199?", "200?", "19??", "20?? or "????", using "?" characters, to be displayed as is to express missing information)
• CRC32 checksums of all image files referenced in the configuration part of the application description
• Thumbnail (path using URL syntax, not absolute but relative to XML file, without path if in same directory/archive, pointing to PNG, GIF or JPEG file with no transparency, no animation, square pixels, and with name consisting of maximum 30 "a-z", "0-9", ".", "-" and "_" characters, having a maximum image size of 160x120 pixels, with at least the width or the height being exactly the maximum size)
• Homepage (URL of the application homepage, with preference to the website of the referenced Entity, if still active; this is not a download link, but the best-possible way to reach the official homepage of the specific application)

Notes:

• There may be one and only one Data Version, UUID and Title field (required fields)
• There may one or more multiple Type categorization elements (at least one is required, more are possible, e.g. a medium may contain both a game and video data, or generic application software and audio tracks); the Amiga prefix is specified as "a-" rather than "amiga-" in order not to inadvertently use possible registered trademarks and in consideration of possible compatible environments such as AROS
• There may be zero or one of all other fields (optional fields); do not use "-" or "" to indicate blank or unknown fields; simply omit the field in that case
• UUID, Type and Year are rigidly defined, whereas the other fields are "free form"
• UUID is a 36-character string as per RFC-mealling-uuid-urn-05.txt (numbers, lower case letters and "-" separators as in "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"), or an OID as per X.208 (as in "1.3.6.1.4.1.xxxx.xxxx", see below about thoughts as to whether it is a viable approach)
• It is desirable that the UUID be uniquely associated to an application, e.g. unlike a CRC32 (or other) checksum of the XML file, the UUID would remain unchanged if the content of the descriptive data set is improved over time; if a description references a different version or variant of an application, then, and only then, a different UUID would be used to identify that (different) application
• It should be avoided that different archives, retrogaming sites, etc. independently issue different UUIDs for the same application; a "first comes wins" procedure should be used, whereby a site can check other sites and use the same identifier, while being free to implement a different description, if so desired; but only if the UUID is identical can differences in content or image file CRC32 be detected, matches be found, etc.
• Type is for possible internal categorization/filtering by application launchers; all other fields are for display to the user exactly as they are (except that Title and Version may be merged with the addition of a single space character separator between them)
• Version should not use "v." or similar introductory "version" part; this field is mostly for system software, productivity, utilities and other applications, not for games or demos, as most games do not have a "version" (e.g. a "II" or "2005" or other sequel indicator in games is considered part of the publication title, not a version denomination).
• Year should be blank only if not applicable, otherwise use tentative style as indicated if a date is to be researched; if range of years (e.g. in copyright notice) use most recent year, not oldest year (listings need to indicate "freshness" of specific version more than "first launch date")
• The character set is ISO 8859-1, as used on the Amiga and other systems (UUID, Type and Thumbnail should use 7-bit ASCII or other subset defined for field, if a specific title introduces a conflict between ISO 8859-1 and ISO 8859-15 or Unicode we may consider the addition of parallel Unicode fields, with ISO 8859-1 transcripts for compatibility)
• Use of proper title style capitalization is encouraged for the Title field (e.g. in English initial caps for all words except for articles and prepositions which are not beginning or end of title, see Chicago Manual of Style or similar, and do not use for other languages which do not use title style capitalization)
• Titles should reflect the publisher's advertised title, or, if that is unknown, a text on the title screen, or, lacking that, some other references (e.g. some strings found in the binaries); modifying the title, e.g. by fetching an initial "The" and placing that at the end is expressly discouraged; the rationale for this is that doing so by hand introduces an arbitrary and uncertain (has it been done? has it not been done? why should it be done only for "The"?) loss of information, whereas the same can easily (and certainly) be done automatically by the front-end or other processor, if so desired.
• Use of a consistent Publisher title is encouraged (ideally, the formal company name as registered, but some publishers may prefer to be referred by their informal name, e.g. "Electronic Arts" instead of "Electronic Arts Inc."); see more on cataloging below.
• Variant is for display purposes only, not for configuration purposes (e.g. emulation software does not rely on this field)
• The SystemROM field is for launcher use only (e.g. to list an entry in a different color if the required system ROM is not present, or to display a helpful message if the user attempts to start the item), not for configuration purposes (e.g. emulation software does not rely on this field)
• It is proposed to place CRC32 information (already being a popular and tested technique to checksum emulation-related images in a filename-independent fashion) in the short description part, rather than in the configuration, or in the extended description, because it has been considered that the short description is the one that would best be used in a collection of hundreds or thousands of descriptions, resulting in a powerful option to not only perform integrity check by the emulation software, but also to do database comparisons on the server side (e.g. if two download sites have different versions of the same disk), and to automatically locate application images on the client side (e.g. putting order through thousands of ADFs on a hard disk)
• Thumbnail specs are strict so that the file itself can be readily used with the least amount of processing on different common systems and networks; 160x120 inspired by several other preview specs; JPEG is not encouraged due to its lossy nature; GIF or palette-based PNG is preferred for thumbnails in up to 256 different colors; true color PNG is preferred for all other thumbnails; JPEG images should not include a preview thumbnail in the thumbnail file itself; use of GIF, PNG or JPEG interlace option ("progressive preview") is optional

Other options:

• Maybe it would be very "powerful" to make placing the CRC32 information for each referenced content file a requirement; that would further back the rationale of placing this information in the short description, and it might result in unexpected applications
• Experimenting with small-size graphics (8, 16 and 32 pixels wide), it was considered that it could be possible, for thumbnail caching and preview purposes, to embed palette-based, non-interlace GIF image data as a few lines of text in the XML itself; about 1500 characters (20 lines at 80 columns) would be sufficient to render a 32-pixel-wide image, which would offer a good compromise of detail and data size (image widths of 8 and 16 pixels do not result in correspondingly smaller data size). Maybe such a field could optionally be used in a future version.

Application Configuration Data Detail

It is envisioned that the Application Configuration part be used by the emulation software to set all configuration options (e.g. Amiga hardware, RAM, ROM, floppy disk insertion order, etc.) required to run a specific application, without changing any user-specific preferences (e.g. input device, etc.)

The Application Configuration part has no information of practical use for purposes of application description e.g. for launcher/loader front-ends.

We would like to propose the use of the following minimal set of Application Configuration fields for immediate practical use by emulation applications:

• Data Version (required field)
• Reference Hardware (the first release, or the specified revision of an exact original unexpanded model, including its exact RAM and ROM configuration)
• Reference Hardware Variant (for Amiga: only "pal" or "ntsc")
• Required Expansions, including replaced parts (e.g. Amiga Fast RAM, Chip RAM, floppy drives, RTG display card, CPU board variation, ROM variation, etc.)
• Reference Software (not necessary for self-booting applications)
• Instructions for default application image "insertion" (e.g. "this disk in drive 0", etc.)
• String variable to be passed to Amiga environment (e.g. when the launcher of the individual application is actually inside an otherwise shared Amiga environment)

Because the emulation software can provide some unique advantages over the original hardware, we furthermore would like to propose a minimal set of fields which would not be applicable in a strict context of references to "real" hardware, but which we believe to have a positive practical impact:

• Reference Hardware Variant Required or Not: "yes" if strict, "no" if the user can set a different configuration option, and the application will still work (e.g. if the user prefers to run in a PAL vs. NTSC environment)
• Maximum Image Loading Speed Supported: "yes" if the application can be launched without problems from an virtually unlimited-speed emulated storage device (e.g. instead of loading from an original-speed emulated floppy drive, a game can be loaded almost instantly); "no" if there are timing-dependencies in this field
• Maximum CPU Speed Supported: "yes" if the application can run without problems (probably with benefits) on faster-clocked CPUs (of the same type as the one specified in the reference hardware); "no" if there are timing-dependencies in this field

When defining the reference hardware, in order to avoid any future discussion about unexpected variations maybe we should define not only a piece of hardware (e.g. "a-500" is an "Amiga 500"), but also the serial number of the actual reference hardware, so that we can refer to that in case of doubt.

For Amiga applications, the following models have been considered (default ROM version still to be defined for each model):

• a-1000: Amiga 1000 (256 kb of RAM)
• a-500: Amiga 500
• a-2000a: Amiga 2000 rev. A (initial release)
• a-2000b: Amiga 2000 rev. B (most popular version)
• a-2000c: Amiga 2000 rev. C (totally different chipset)
• a-3000: Amiga 3000 (initial release, ROM on file)
• a-500+: Amiga 500+
• a-600: Amiga 600
• a-4000: Amiga 4000
• a-1200: Amiga 1200
• a-cdtv: CDTV
• a-cd32: CD³²

Unless expanded, all devices start with one floppy drive, except for the CDTV and CD³², which have none. Should any of these be HD by default (e.g. in the A3000)?

When choosing the default "canonical" system, we should probably be practical: for the A1000 it would probably not make sense to specify a 1.0 ROM, as that was both replaceable and replaced, but rather maybe it should be 1.1. Similarly, maybe if we were to decide on the default A2000 ("a-2000"?), that should probably be the B revision, not the first one.

Should we also define from the start variations such as assembled systems, e.g. A1500, A2500, A3000T and A4000T? Doesn't cost much... And just for completeness we could include third-party solutions such as Draco, Casablanca, Arcadia Multi Select, Access and Boxer.

Should we define a "required" configuration and optionally an "ideal" one (or "best" and "minimum")? Probably yes. For example, the game Mind Walker runs perfectly on a 1.1 Amiga system, but also runs (with some visual imperfections) on more common 1.2 and 1.3 systems. The game Arcade Pool runs best on an AGA system, but it also works on an OCS Amiga. Is an optional second configuration enough, or should the system allow for unlimited fallback options? What about multiple combinations of fallback (ROM version, chipset, etc.), each handled in different ways by the application? Having collected a lot of feedback on this topic, it is currently believed that a dual configuration approach serves well even more complex needs, while at the same time limiting complexity.

About peripherals (variations to canonical hardware), tags could include:

• ROM chip version (as if the ROM were to replace the default ROM specified for the machine)
• Additional Slow, Chip, Fast, Zorro III, in kb (or should it be total, overriding the RAM defined in the default canonical system?)
• Number of additional DD floppy drives
• Number of additional HD floppy drives
• Picasso II RTG display card (2 MB RTG RAM if not specified otherwise)
• Total RTG RAM, replacing default RAM set for RTG board (or additional? consistency with system RAM would be desirable)
• A2620 68020 expansion board (revision? RAM?)

How to define input device requirements (joysticks, etc.)? Very little in-house experience here.

Other peripherals that are commonly required by Amiga application software, even if not yet emulated (e.g. PPC board, VideoToaster)? Define a full list (as in "Sara's BBS"), e.g. to associate it to Amiga original driver disks?

Should we allow for the possibility to "remove" an item from a reference configuration? E.g. to remove the floppy drive from an A3000. But maybe this is overkill at this stage, and not at all practical for the goal of defining an existing set of applications. Items can be added and defined as necessary.

For the reference software configurations, the following standard configurations could be a starting point:

• Workbench 1.0 floppy (NTSC only) in DF0
• Workbench 1.1 floppy (NTSC) in DF0
• Workbench 1.1 floppy (PAL) in DF0
• Workbench 1.2 floppy in DF0
• Workbench 1.3.0 floppy in DF0
• Workbench 1.3.4 floppy in DF0 (almost identical to Amiga Forever 1.3 environment boot disk)
• Amiga Forever 2.X environment (Amiga Forever 2006 and higher)
• Amiga Forever 3.X environment (Amiga Forever 2005 and higher)

For fallback purposes, when unavailable 1.x boot floppy disks are required, the emulation software could maybe, with some user approval, try to use floppy disks in increasing version number order first (e.g. using 1.2 instead of 1.1 NTSC or PAL, or 1.1 NTSC instead of 1.0), and then, if this still fails to find a disk image, try to decrease the version number (e.g. using 1.3.0 instead of 1.3.4, or 1.1 NTSC instead of 1.2). This appears to be practical, as for example 1.3 (the more commonly available Workbench image, as included in all versions of Amiga Forever) is almost identical with 1.2, and the 1.3.x are also mostly compatible between each other. A message could be displayed indicating that a different software environment was loaded, listing both the desired one and the used one, and perhaps offering to OK or Cancel.

Extended Application Description Data Detail

It is envisioned that the Extended Application Description part be used for additional cataloging, description and listing purposes. This part may be "overkill" for a simple application launcher front-end, however it may be very useful to store and exchange data between emulation sites, etc.

This section is deliberately kept short for now, but fields that come to mind that could be in this section include:

• Additional application notes, e.g. description, individual application developers (e.g. with specific tagging for credits to code, graphics, music, special effects), etc., with texts in multiple languages (tagged as per  ISO 639-1, using only two lower case letters; issue: use ISO 8859-1 for fully supported languages and Unicode for everything else?)
• References to screenshots and other files (all with CRC32 included in this section, not in the compact description set)
• Additional fields mapping to TOSEC-defined fields and to fields used by various well-known existing sites (e.g. HOL has both a short name and a full name for the publisher's name, we could use the common or short name in the compact description, and add the extended name here, if different)

Thoughts:

• Would this be a good point to begin to uniquely catalog and assign OIDs to all legal entities (companies, persons) involved in an application (e.g. Amiga games), as well as the applications themselves (use OID instead of UUID)? This would allow for further refinements to company names, support for changing entity names, etc., while allowing for the continued and unequivocal identification of the same.

Exploring Possible First Steps into Practice

We would like to combine theory and practice in a first implementation specific to Amiga games and demos (which is neutral enough not to exclude other platforms and applications).

It is envisioned that possible scenarios could include:

• Use of a single ZIP archive with a flat set of files (no directories) contains the XML file with at least the compact description and the configuration information (.xml or other specific suffix? default file name?), the thumbnail, and the ADFs or other image file(s)
• A website offers one interface (using the UUID as the argument, e.g. in the URL) to directly access such a ZIP file, and one interface to open the download page with a description; a site may choose to implement either one or the other or both methods; the direct-to-file interface can be combined with a way to open an additional web page at the same time, so that there is no loss of banner information (e.g. Windows Media files can similarly associate websites to movie previews, and one is opened while the other is played)
• A website can offer a collection of multiple compact descriptions in the same XML file, describing a partial or total catalog of titles it has available
• An application like a front-end in Amiga Forever or in WinUAE (or Software Director, made available to all) can compare available online titles with local descriptions, present to the user a front-end listing a title as installed or available, offer a one-click option to download and auto-install a title, and offer to run the title in the same front-end
• Downloaded applications should be organized in a central location, tentatively agreed on at least on Windows (e.g. see 15-101 and 15-112)
• The extracted ZIP archive content, or content organized from other sources (e.g. automatically organized by assembling files found on the disk) should be stored in a single directory (issue: this being a combination of various images, not just ADFs, should perhaps be stored under a directory referencing its self-describing and self-organized nature, more than the type of image files, in its name, which makes us wonder how to name what we are talking about :-)
• In order to avoid undesired collisions, and to detect intended collisions, it could be possible to create subdirectories (one for each application) named after the UUID; this would be rather user-unfriendly at first look, but considering that a way has to be found to automatically manage a system with no undesired namespace collisions, and that the front-end offers a way to open the directory associated to an application, and that any OS offers a Search option, it may after all be a good compromise

More

More to follow. But feedback is essential!

Related Links

• Use of XADML Files in Amiga Forever
• RetroPlatform OIDs
• Amiga Configuration Identifiers