View source for User:Fæ/Project list/NYPL

== Introduction ==
[[File:God seated in the center, surrounded by a circular serpent biting its own tail as a symbol of the returning cycles of the year (NYPL b12455533-426176) detail.jpg|thumb|right|Detail from  a 13th-century illuminated manuscript held by the NYPL, showing God at the center of a calendar.]] 
In December 2015, the NYPL announced an improvement to how images were licensed for public reuse, and made 180,000 available with public domain licenses. This upload project mirrors these images on Commons and creates image text pages based on the metadata available from the NYPL {{w|Metadata Object Description Schema|MODS}} records.

Though the NYPL release is for 180,000 artifacts<!-- U.S. spelling-->, the number of images uploaded to commons may be larger as some documents have multiple pages, others have been uploaded in the past at lower resolutions under different names, and where full size jpeg versions are available as well as the TIFF originals, both are to be uploaded. As of October 2016, the total number of NYPL images as part of this project was 222,000 images uploaded with total file storage needed of more than 8,000 gigabytes (8 {{w|terabytes}}); see [https://quarry.wmflabs.org/query/10552 quarry report].

;Links
* A summary of the collections are at http://www.nypl.org/research/collections/digital-collections/public-domain.
* A visualization page can be found at http://publicdomain.nypl.org/pd-visualization.
* The NYPL runs an open {{w|API}} which is described at http://api.repo.nypl.org. Accessing it needs registration.

__TOC__

== Collections batch uploaded ==
The numbers below represent the latest figures for how many files are in the category, in some cases files have been diffused since upload into sub-categories which are not counted.
<gallery style='background:lightgreen' mode="nolines" caption = "Sample gallery and notes for NYPL collections uploaded">
Pyramids of Major Dade and 107 of his Men, who were massacred at Fort Dade, Florida, by Osceola and his Band (NYPL b11707414-G90F138 034ZF).tiff|{{User:Fæ/pagesincat|NYPL The Miriam and Ira D. Wallach Division of Art, Prints and Photographs: Photography Collection}}

When a nigger makes a hundred, ninety-nine goes on his back (NYPL Hades-609544-1256662).jpg|{{User:Fæ/pagesincat|NYPL American popular songs}} (the majority of files in this collection get rejected by the WMF API as having bad tiff info, it is unclear why, so this upload has been restricted to full size NYPL jpegs. See [[Phab:T124662]].)<!-- 35301010-c58d-012f-1196-58d385a7bc34 'NYPL American popular songs' -->

A_south_east_view_of_the_great_town_of_Boston_in_New_England_in_America_(NYPL_Hades-250999-465401).jpg|{{User:Fæ/pagesincat|Emmet Collection of Manuscripts}}

Buckskin_Charlie,_Sub_Chief_of_the_Utes,_Indian_(NYPL_b12647398-62164)_(cropped).jpg|{{User:Fæ/pagesincat|Detroit Publishing Co.}} (136 jpeg files previously existed in this category and 1,297 jpegs from the Library of Congress at [[:Category:Photochrome pictures from Detroit Photographic Company]], refer to a previous proposal at [[Commons:Batch_uploading#Detroit_Publishing_Company_at_LoC]])

Frontispice pour l'Illustration nouvelle (NYPL b14506647-1128825).jpg|{{User:Fæ/pagesincat|Samuel Putnam Avery Collection}}

Miniature of Annunciation. Figure of monkey and cat playing a musical instrument in border. Initials (NYPL b12455533-426020).jpg|{{User:Fæ/pagesincat|NYPL Manuscripts and Archives Division}}

To the Hone. Ino. Hancock Esqre. president of ye Continental Congress, this map of the seat of Civil War in America.. (NYPL Hades-118253-54199).tif|{{User:Fæ/pagesincat|NYPL maps}} this is a refresh, see [http://tools.wmflabs.org/catscan3/catscan2.php?language=commons&project=wikimedia&categories=NYPL+maps%0D%0AImages+uploaded+by+F%C3%A6&ns%5B6%5D=1&after=20160101&ext_image_data=1&file_usage_data=1&doit=1 catscan] ~3,335 map images in this tranche.

Mazurier, rôle de Joko (NYPL b12148947-5236176) (cropped).jpg|{{User:Fæ/pagesincat|Jerome Robbins Dance Division}}

Theben (Thebes)- Ansicht der Kolosse Amenophis (Amenhotep) III. (Memnon) (NYPL b14291191-37597).jpg|{{User:Fæ/pagesincat|Monuments from Egypt and Ethiopia to the drawings of the kings of Prussia, Friedrich Wilhelm IV to these countries sent by his Majesty and ausgefuhrten in the years 1842-1845}}

Le rendez-vous des chats (NYPL b14261587-483635).jpg|{{User:Fæ/pagesincat|NYPL The Miriam and Ira D. Wallach Division of Art, Prints and Photographs: Print Collection}}

Army and Navy Nights (NYPL b18358405-5150558).tiff|{{User:Fæ/pagesincat| R. H. Burnside Collection }}

Thèbes. Hypogées. 1-3.6-8. Peintures dessinées d'après des enveloppes de momies; 4.5. Détails d'une frange d'une toile rayée, trouvées sur des momies (NYPL b14212718-1267989).jpg |{{User:Fæ/pagesincat| Description de l'Egypte}} ~200 versions pre-existing

Mid-Manhattan, Reference books, readers in background (NYPL b11524053-1252852).tiff|{{User:Fæ/pagesincat|New York Public Library Visual Materials}}

I'm All Alone - Midland Beach, Staten Island ((lady in early bathing costume poised to dive into water, superimposed on painted water scene) (NYPL b15279351-104487).tiff | {{User:Fæ/pagesincat| Staten Island post cards}}

The world in planisphere (NYPL b14428893-1503411).jpg|{{User:Fæ/pagesincat|Lawrence H. Slaughter Collection of English maps, charts, globes, books and atlases}}

Washington's grand entry into New York, Nov. 25th, 1783. Scene junction 3d Ave. and the Bowery (NYPL Hades-1784723-1650645).jpg|{{User:Fæ/pagesincat| Eno collection of New York City views}}

Ananas fructu ovato, carne albida - Carduus Brasilianus, foliis Aloes. (Pineapple) (NYPL b14444147-1125000).jpg|{{User:Fæ/pagesincat| Hortus Romanus juxta systems Tournefortianum paulo}}

Cannonball Titcomb, Pitcher (NYPL b13537024-55787).jpg|{{User:Fæ/pagesincat| A. G. Spalding Baseball Collection}}

</gallery>

<div style='{{divstylered}}'>
;Others
#{{User:Fæ/pagesincat|Buttolph collection of menus}}
#<!-- 25 de Mayo (NYPL b14896507-83707) (cropped).jpg -->{{User:Fæ/pagesincat|Vinkhuijzen collection}} (total reduced as files are moved into subcategories)
#{{User:Fæ/pagesincat|Theodorus Bailey Meyers Collection}}
#{{User:Fæ/pagesincat|Samuel J. Tilden Collection}}
#{{User:Fæ/pagesincat| Picturesque Palestine, Sinai, and Egypt (NYPL scan)}}
#{{User:Fæ/pagesincat| Pacific pursuits postcards}}
#{{User:Fæ/pagesincat| NYPL collection of Atlases, gazetteers, guidebooks and other books}}
#{{User:Fæ/pagesincat| Birds of America, from drawings made in the United States and their territories}}
#{{User:Fæ/pagesincat| Traité des arbres et arbustes que l'on cultive en France en pleine terre}}
#{{User:Fæ/pagesincat| A collection of the dresses of different nations, antient and modern}}
</div>

Other collections are not listed here and are mostly under 500 images in size.

Initial uploads relied on {{tlx|information}} and later uploads apply the ingestion template {{tlx|nypl}}.

The uploads are picked from http://publicdomain.nypl.org/pd-visualization/

== Numbering ==
Filename scheme is:
 <title> (NYPL <collectionid>-<imageid>).tiff
Where the title is as provided by the NYPL but truncated to sub-215 characters. The collectionid uses NYPL b-number or OCLC reference if not available, or the Hades legacy number if neither of those are in the metadata. Jpeg versions are identical apart from the extension.

The standard credit template of {{tlx|NYPL-image-DigitalID}} is used, which adds a parent bucket category automatically.

== Automated checks and validation ==
As of June 2016, Fæ's processes for uploading new images involves several robust customized checks using pywikibot-core, and additional modules available in Python. The order of checks is based on the lag they may create in processor time and the amount of data to be transferred over the internet. Any image that fails one of these checks is skipped on the first failure, unless another action is specified.
# There is a text search for the intended Commons filename to ensure it has not been created previously.
# The metadata for the image page is extracted from the NYPL MODS data. This in turn provides links to any pre-existing high resolution jpeg versions of the TIFFs. The JSON structure of the MODS data from NYPL is inconsistent, so a heuristic approach has been taken which may not be ideal for capturing all useful metadata as unexpected data structures causing errors when parsing are normally skipped and the image text page created without those elements.
# Topics included in the MODS metadata are checked for matching Commons category names. See [[#Categorization]]
# Commons is searched for text matches to the NYPL ID numbers. If there are any matches then the file has most likely been uploaded before, though possibly with a different set of EXIF data.
# Past deletions for the suggested file name are searched for. This avoids re-uploads of files that were deleted for being 'mostly blank' or similar; there is a good faith presumption that any past deletion is valid.
# Commons is searched for matches to the local SHA1 value. Unfortunately the NYPL metadata does not provide SHA1 values which could be used in advance of downloading a file.
# Immediately after upload the image text page is checked to see it is greater than 300 characters, if not then the text page is regenerated. This is an outstanding rare bug for uploads and may be related to WMF server drop-outs or more general internet connection problems. Refer to [[Phab:T113878]].

=== Image page galleries ===
Post-upload a routine is manually run on collections where galleries of associated files will add value. The "association" is worked out automatically by searching Commons for the artifact UUID. For example [[:File:Stone, Thomas (NYPL b11868620-5338054).jpg]] relates to a gallery on the NYPL source site with four images. As both the TIFF and jpeg files have been uploaded to Commons, with the jpeg versions providing useful cropped versions of the scans, there are eight files related to each other, so the example image page cross-links by using a gallery of seven thumbnails. The gallery makes it much easier to navigate between pages of a document while still making it easy for Wikipedia contributors to select a single page or plate to illustrate an article.

=== Blank pages ===
{|class=wikitable width=300px style="float:right"
|-
|[[File:DINNER (held by) THE BRUNSWICK (at) "BOSTON, MA" (HOTEL) (NYPL Hades-269422-4000000184).tiff|x150px]]
|Blank page for an 1881 4-page menu of The Brunswick Hotel, in the Buttolph collection. Due to damage, unlikely to be auto-detected as blank. See [https://commons.wikimedia.org/w/index.php?search=168bfe20-c530-012f-dae2-58d385a7bc34&title=Special%3ASearch search].
|-
|[[File:Col. Eleazer Oswald (NYPL Hades-280105-EM3431).tiff|x150px]]
|| Page identified as not blank, as none of the central 4 zones is blank. The image would have been detected as 'mostly blank' without the bulls-eye test.
|}
;[[:Category:NYPL (mostly blank)]]
[[File:Internet Archive mostly blank example.jpg|thumb|left|x180px|Illustration of the 'chessboard' breakdown of images.]]
Some blank pages are a 'feature' of parts of the archives. For example the scans of music scores are likely to include a blank page after the cover page, and the scans of stereograms include the backs and though some have interesting descriptions or indexes, many others are blank.

It is possible to attempt to auto-detect these and mark them for review, the same programme used for [[User:Fæ/Project_list/Internet_Archive#Blank_pages]] has been adapted to check the NYPL images. This can be run as a semi-automated housekeeping task on demand, depending on how significant an issue it seems for the community.

In addition to the "mostly blank" detection routine used for Internet Archive images, due to a number of very small portrait images in the centre of book plates, the middle four squares of the chessboard grid are checked to ensure that at least two of the four are blank in order for the image to be considered mostly blank.
{{clear}}

== Jpegs ==
{| class = wikitable style = "float:right;width:280px;text-align:center;font-size:8pt"
|-
|[[File:Charles Carroll, barrister, member of the Continental Congress (NYPL b12349185-420062).jpg|x150px]]<br>Jpeg 2.2 MB
|[[File:Charles Carroll, barrister, member of the Continental Congress (NYPL b12349185-420062).tif|x150px]]<br>TIFF 22.7 MB
|-
|colspan=2|Portrait of Charles Carroll both in jpeg and TIFF formats, as taken from NYPL archives but render differently in thumbnails on Commons.
|-
|[[File:Monsieur Balon (NYPL b12145898-5073648).jpg|x150px]]<br>Jpeg 4.8 MB
|[[File:Monsieur Balon (NYPL b12145898-5073648).tiff|x150px]]<br>TIFF 173 MB
|-
|colspan=2|Drawing of Monsier Balon, 1700. The jpeg version has been cropped and rotated by the NYPL curator using the full TIFF scan, so at the time of upload is already suitable for illustrating Wikipedia articles.
|}
As well as the master TIFF, a jpeg version will be loaded if a ''full size'' derivative is identified in the metadata. For many collections the only jpegs available are sub-800px wide and not worth loading.

The advantage of having a jpeg is that they make better (sharper) thumbnails at the current time, and are easier to view within Commons, while the TIFF needs to be downloaded to view it {{w|Lossy compression|losslessly}} (this may improve in the future). Some of the NYPL collections offer more value in the jpeg versions as they have been cropped and rotated by the curators.

Where crops are needed, such as trimming off redundant scanned background, blank page margins or colour swatches, the jpeg can be cropped using [[COM:CropTool]]. Best practice would be to leave the TIFF version untouched as the archive reference image, however cropped derivatives of the TIFF can be uploaded if there is no original high resolution jpeg from NYPL.

After an upgrade to Pywikibot-core in June 2016, made necessary by the WMF rejecting http API queries, it was found that the upload module did not seem to behave well in response to exceptions. This may result in jpeg versions being delayed for an indefinite time. Refer to [[#T138206]].
{{clear}}

== Galleries ==

Many of the documents in the NYPL collections have multiple pages. The {{w|Universally unique identifier|UUID}} link in the image description provides an easy way of finding all "live" pages and variations of the same document. In addition there is a post-upload housekeeping process to create gallery tables placed at the '''Other versions''' parameter in the information box for each image, see {{w|Help:Gallery tag}}. This cross-links each page with thumbnails for all the other pages with the same UUID, giving an easy visual way to navigate around a document. Where both TIFF and jpeg versions of the same page exist, these will be shown as duplicate images in the gallery, though as the filename is shown when the user floats their mouse over the thumbnail, the user has a choice over which type of file to jump to.

When images are deleted, at the current time this is only likely to happen for mostly blank pages with little educational value, the gallery will show a blanked space for the link. The housekeeping routine can be re-run to update all galleries including removing deleted files, if there are sufficient numbers for it to be worthwhile.

== Categorization ==
The NYPL metadata includes subject / topics which are checked to see if there are matching Commons categories, and added if not marked as {{tlx|CatDiffuse}} or {{tlx|Categorise}} categories. The full list of topics is added to the image page text information.

As the NYPL metadata has an inconsistent JSON format, the use of topics cannot be relied on for any particular image or collection.

As of June 2016, the family tree of categories for the NYPL collections is not ideal. Images may appear in all of a specific collection category, a NYPL department category and the template based bucket category. It could be that the auto-generated department categories might be mass removed where specific collection categories exist, on the basis that a collection is always within a department.

From July 2016, the batch upload process started to recognize where categories use {{tlx|category redirect}}, and skip any category matching that condition. This avoids the situation where the images are moved automatically to a category already marked with diffusion templates and may cause a "flooding" problem.

Examples:
* [[:File:Paraqvaria Vulgo Paragvay Cum adjacentibus (NYPL b14467885-1505059).jpg|Paraqvaria Vulgo Paragvay Cum adjacentibus (NYPL b14467885-1505059).jpg]] has a single NYPL topic of "history". The matching category was checked but uses the CatDiffuse template and so is ignored as a suggestion. The image page still lists history in the Topics field.
* [[:File:Army and Navy Nights (NYPL b18358405-5143306).tiff|Army and Navy Nights (NYPL b18358405-5143306).tiff]] has topics "Theater; New York (State); New York; Theater; Production and direction". ''Theatre'' and ''New York'' use the Categorise template and are skipped, while the other two topics have no category matches and are ignored.

== Use of [[COM:GWT|GLAMwiki toolset]] ==
The cost/benefit ratio of crafting XML output so that GWT could present similar image pages and categorization is not really worth the "calendar time" saving, as this would mean carefully reviewing each collection to handle JSON mapping oddities. Similarly, avoiding a "big bang" upload process means that the community has a more realistic period to discover the new uploads and provide feedback on quality issues.

The GWT was used for uploading some TIFFs over 100MB in filesize, but this became unnecessary after changing from Pywikibot ''compat'' to the ''core'' version which includes the ability to use chunked uploading and files up to 1GB in size. Based on past experience, home broadband may effectively limit uploads to 8GB per month, principally due to slow upload speeds, a magnitude slower than what is possible with GWT. However as a slow upload process gives more time for feedback and corrections, this may end up being a key additional benefit.

The above method has been superseded by white-listing of url transfers for both TIFF and jpeg images from the nypl.org domain. This enables fast uploading of even the 100MB+ files as they are no longer reliant on my home broadband connection. There are still double checks on SHA1 matches and post uploaded double checks on the image pages, but the process is a magnitude faster.

== Multiprocessing ==
[[File:Pywikibot multiprocessing uploads.png|thumb|600px|right|Three image upload threads with one finishing after 10 minutes of upload connection time. "J49" was the unique local task name, with "223" showing this was image 223 out of the NYPL collection of several thousand.]]
From 19 May 2016, some of the uploads started to use [https://docs.python.org/2/library/multiprocessing.html multiprocessing]. This enables the uploading of several files in parallel. The Python code works as normal, going through the collection linearly, but then spawns processes to handle each upload.

After trials, the parallel uploading was limited to 2 files at the same time for the same collection. Though multiprocessing ensures that there is no gap between uploading files while parsing source metadata, there was no apparent speed benefit in having more than a couple of files uploading in parallel. The reasons for this are likely to be a combination of (a) there are automated throttling processes at the Wikimedia server end, (b) it is likely that automated throttles apply at the NYPL source, and (c) the files are reliant on a home broadband connection with a fixed upload bandwidth limit.

For the NYPL batch upload, the relevant multiprocessing calls use the module Process and a custom fuction ''processup'' which handles the Commons upload call. Here's the relevant call which creates the parallel processing thread and sticks the reference to it in an ''threads'' array which then is later used to track the pool of active threads and waits for one to complete whenever the maximum desired number is reached before adding the next:
<syntaxhighlight lang=Python3 line >
t = Process(target=processup, args=(source, filename, d, acount, count, totalpages, perpage, filenameid, tname, comment + " " + tname, localsize, ), name=tname)
threads.append(t)
t.start()</syntaxhighlight>

== Parsing the NYPL API (with Python) ==
The NYPL API service requires a login which you set up by providing an account name and password on the website http://api.repo.nypl.org/. In response the site provides you with a token for API calls which is valid for a maximum of 10,000 requests in a day (if you hit this, you are probably using calls very inefficiently!). When pulling information from the API, Python needs to put an authorization code in the url request header. This looks like <syntaxhighlight lang=python3 inline>headers = { 'User-Agent' : 'Mozilla/5.0' , 'Authorization': 'Token token="abcdefghijklmn"'}</syntaxhighlight> where the token is the long code that NYPL provided and ''headers'' would be passed in calls like <syntaxhighlight lang=python3 inline>http.urlopen('GET', u, preload_content=False, headers=headers)</syntaxhighlight>. ''I happen to be using [https://pypi.python.org/pypi/urllib3 urllib3] to handle https, but that's not essential for the NYPL calls to work.''

Here's a cut & paste from a terminal window where my parser is pulling out relevant information for an NYPL image and ends up creating [[:File:Italy._Kingdom_of_the_Two_Sicilies,_1817-1819_(NYPL_b14896507-1608945).tiff|Italy._Kingdom_of_the_Two_Sicilies,_1817-1819_(NYPL_b14896507-1608945).tiff]] as part of the [[:Category:Vinkhuijzen collection|Vinkhuijzen collection]] upload:
<pre style="background-color: #dee; border: 1px solid #0000CC; margin: 0.5em; padding: 0.5em;">
 ID_Hades struc ID (legacy)  1098453
 ID_uuid  98b9f420-c54a-012f-b4ed-58d385a7bc34
 uuid  510d47e3-f235-a3d9-e040-e00a18064a99
 title  Italy. Kingdom of the Two Sicilies, 1817-1819.
 url  http://digitalcollections.nypl.org/items/510d47e3-f235-a3d9-e040-e00a18064a99
 topics  
 ID_Hades Collection Guide ID (legacy)  206
 filename  Italy. Kingdom of the Two Sicilies, 1817-1819 (NYPL b14896507-1608945).tiff
 imageID  1608945
 note  Italy. Kingdom of the Two Sicilies, 1817-1819.
 source  http://link.nypl.org/oCcX5CERRragkGJruYT_Ywr
 ID_NYPL catalog ID (B-number)  b14896507
 date  
 ID_CATNYP ID (legacy)  b6535738
 modslink  http://api.repo.nypl.org/api/v1/items/mods/510d47e3-f235-a3d9-e040-e00a18064a99
 ID_RLIN/OCLC  45057766</pre>
The ''filename'' is being generated using [[#Numbering|the naming convention]] using the NYPL title and best NYPL unique identities for the artifact plus the ''imageID'' which is then unique to the page level within an artifact (i.e. document). Note that no ''date'' or ''topics'' have been found for this artifact.
The ''modslink'' provides most of this information but requires {{w|JSON}} parsing. The parsing needs to be done defensively, in practice the data scheme used may vary between collections, for example the use of 'note' fields to contain a useful description may be absent and in this example is being replaced by the title text. The NYPL unprocessed source looks as below:
<syntaxhighlight lang = JSON style="border: 1px dashed silver;">
{"nyplAPI":{"request":{"uuid":{"$":"510d47e3-f235-a3d9-e040-e00a18064a99"}},"response":{"headers":{"status":{"$":"success"},"code":{"$":"200"},"message":{"$":"ok"}},"mods":{"version":"3.4","schemaLocation":"http://www.loc.gov/mods/v3 http://www.loc.gov/standards/mods/v3/mods-3-4.xsd","titleInfo":{"supplied":"no","usage":"primary","title":{"$":"Italy. Kingdom of the Two Sicilies, 1817-1819."}},"name":{"authority":"","type":"personal","valueURI":"","namePart":{"$":"Vinkhuijzen, Hendrik Jacobus"},"role":{"roleTerm":[{"authority":"marcrelator","type":"code","valueURI":"http://id.loc.gov/vocabulary/relators/col","$":"col"},{"authority":"marcrelator","type":"text","valueURI":"http://id.loc.gov/vocabulary/relators/col","$":"Collector"}]}},"typeOfResource":{"$":"still image"},"language":{"objectPart":"","languageTerm":[{"authority":"iso639-2b","type":"code","valueURI":"http://id.loc.gov/vocabulary/iso639-2/eng","$":"eng"},{"authority":"iso639-2b","type":"text","valueURI":"http://id.loc.gov/vocabulary/iso639-2/eng","$":"English"}]},"subject":{"authority":"lcsh","valueURI":"http://id.loc.gov/authorities/subjects/sh2008107806","topic":[{"authority":"lcsh","valueURI":"http://id.loc.gov/authorities/subjects/sh85139693","$":"Military uniforms"},{"authority":"lcsh","valueURI":"http://id.loc.gov/authorities/subjects/sh85061212","$":"History"}]},"identifier":[{"displayLabel":"CATNYP ID (legacy)","type":"local_catnyp","$":"b6535738"},{"displayLabel":"RLIN/OCLC","type":"local_other","$":"45057766"},{"displayLabel":"NYPL catalog ID (B-number)","type":"local_bnumber","$":"b14896507"},{"displayLabel":"Hades Collection Guide ID (legacy)","type":"local_hades_collection","$":"206"},{"displayLabel":"Hades struc ID (legacy)","type":"local_hades","$":"1098453"},{"type":"uuid","$":"98b9f420-c54a-012f-b4ed-58d385a7bc34"}],"location":[{"physicalLocation":[{"authority":"marcorg","type":"repository","$":"nn"},{"type":"division","$":"General Research Division"},{"type":"division_short_name","$":"General Research Division"},{"type":"code","$":"GRD"}],"shelfLocator":{"$":"8-MMEH (Vinkhuijzen collection of military uniforms)"}},{"shelfLocator":{"$":"8-MMEH (Vinkhuijzen collection of military uniforms) vol. 401"}},{"physicalLocation":[{"type":"division","$":"General Research Division"},{"type":"division_short_name","$":"General Research Division"},{"type":"code","$":"GRD"}]}],"relatedItem":{"type":"host","titleInfo":{"title":{"$":"Italy. Kingdom of the Two Sicilies, 1817-1819."}},"identifier":[{"type":"uuid","$":"938f3a70-c54a-012f-08ac-58d385a7bc34"},{"type":"local_hades","$":"1097395"}],"relatedItem":{"type":"host","titleInfo":{"title":{"$":"Italy"}},"identifier":[{"type":"uuid","$":"bd07a6c0-c546-012f-6cf8-58d385a7bc34"},{"type":"local_hades","$":"773362"}],"relatedItem":{"type":"host","titleInfo":{"title":{"$":"The Vinkhuijzen collection of military uniforms"}},"identifier":[{"type":"uuid","$":"51894d20-c52f-012f-657d-58d385a7bc34"},{"type":"local_hades","$":"269277 local_catnyp"}]}}}},"rightsStatement":{"$":"We believe that this item has no known US copyright restrictions. The item may be subject to rights of privacy, rights of publicity and other restrictions. Though not required, if you want to credit us as the source, please use the following statement, \"From The New York Public Library.\" Doing so helps us track how our collection is used and helps justify freely releasing even more content in the future."}}}}
</syntaxhighlight>

== Phabricator tickets and unresolved failures ==
=== Miscellaneous tickets ===

* [[Phab:T137687]] added the domain ''images.nypl.org'' to the Commons url whitelist. This enabled url uploading of jpeg versions of images in addition to the TIFFs which come from ''link.nypl.org''.
* [[Phab:T140075]] (July 2016) WMF ops requested to limit uploads as my account seemed to account for unexpectedly large data spikes. Ops were planning on ~140GB/day but my NYPL uploads alone were hitting over 220GB/day. Planned new hardware to go online this month will make higher data rates easier to accommodate and this irrelevant.

=== Stashedfilenotfound ===
{{anchor|T138206}}
This is an error introduced during the transition to Pywikibot-core. It makes it impossible to use url transfers to upload large TIFFs where the jpeg version already exists (the error to be ignored is termed "exists-normalized" by the Commons API). At the current time there is no easy work-around. For jpeg versions to be uploaded, these are still being uploaded from a local copy without using "chunking", in order to get around the problem, but this would be an unrealistic solution for TIFFs which may be of the order of 200MB in size. As a result of this problem, there may be NYPL files that have been uploaded in jpeg format for which the TIFF version has been skipped.

This bug was first raised on Pywikibot in March 2016. Refer to [[Phab:T129471]] and [[Phab:T138206]].

=== Multi-sheet problem (done) ===
<div style='background:lightgreen;padding:0.5em;border:1 silver solid'>
{| class=wikitable style="float:right;background:lightgreen;text-align:center" width=330px
|-
|[[File:Our Alaskan sisters up in the Klondike country (NYPL b11707251-G89F309 011F).tif |150px|]][[File:Our Alaskan sisters up in the Klondike country (NYPL b11707251-G89F309 011B).tiff|150px]]
|-
|<small>Example of UUID with multiple images, front and back of a stereogram, in this case with a description on the back.</small>
|}

Example item query: http://api.repo.nypl.org/api/v1/items/449d4220-c58d-012f-dcd3-58d385a7bc34
First page uploaded: [[:File:The daisy (NYPL Hades-446489-1152945).jpg]]

The default assumption is that the UUID matches to an image, however multiple images (like the sheet music above) have multiple ImageIDs to one UUID. The group can be pulled out of the imageLink matches as a list to loop on.

{{done}} Though it would be nice if the images cross-linked using ''other_versions'', clicking the UUID will show all images relating to the object on Commons. A large example can be found at [[:Category:The Automobile Club of America Tour Book]]. </div>

=== Colour profile problems ===
{{tracked|T123210}}
Some TIFFs have types of colour profile that result in badly presented images using Firefox, but may display correctly using Chrome. The colour profile can be stripped or replaced in the TIFF. As changing the file means that checksum (SHA1) duplicate searches will fail, it is best if the original image from the NYPL archives is uploaded and then overwritten.

If a profile detection method can be devised, it may be possible to automate this correction.

A sample of images have the first 500 characters from the colour profile included on the image text page as a hidden comment so that different types of profile can be searched out using source searching. For example matches to [https://commons.wikimedia.org/w/index.php?search=insource%3A%2F_HapplscnrRGB+Lab%2F&title=Special%3ASearch "_HapplscnrRGB Lab"] display badly in Firefox v43 (the current version at the time of writing). TIFF colour profile data is not available through the Commons API, the displayed EXIF data, or any other methods.

===Bad TIFF===
{{tracked|T124662}}
Files where on an attempted upload the API returns 'tiff_bad_file' have yet to be explained. It could be that for some NYPL files, the TIFF compression is in an unusual format. These files cannot be uploaded without being transcoded, so are skipped in this batch project. It is worth noting that for the example under 100MB, the custom uploader uses the Python Image Library to query tiffinfo and works successfully, it is the use of a tiffinfo command at the WMF server side that falls over.
<div style="margin-left:1em;padding:1em;border:2px silver solid;background:lightyellow">
;Example:http://digitalcollections.nypl.org/items/510d47dd-ea5c-a3d9-e040-e00a18064a99 ''All aboard for Podunk''
<nowiki>{u'servedby': u'mw1125', u'error': {u'info': u"This file did not pass file verification: The uploaded file contains errors: tiffinfo command failed: '/usr/bin/tiffinfo' '/tmp/r0fzoW' 2&gt;&amp;1", u'*': u'See http://commons.wikimedia.org/w/api.php for API usage', u'code': u'verification-error', u'details': [u'tiff_bad_file', u"tiffinfo command failed: '/usr/bin/tiffinfo' '/tmp/r0fzoW' 2&gt;&amp;1"]}}</nowiki>
</div>

==Reports==
These are reports created using the GLAM dashboard run by Faebot. The reports include the 80,000 lower resolution jpeg images successfully loaded from the NYPL in 2008 by [[User:Dcoetzee|Dcoetzee]], which distorts the results:

{{/index}}
===Volunteers===
{{/volunteers}}
===Largest===
{{/largest}}
===Popular categories===
{{/popular_categories}}
===Improvement===
{{/improvement}}
[[Category:Images from the New York Public Library]]