A FIVE STAR BOOK

I just finished reading Whisper of the Wicked saints. I fell in love with the caracters. I expected an average romance read, but instead I found one of my favorite books of all time. Just when I thought I could predict the outcome I was shocked ! The writting was so descriptive that my heart broke when Julia's did and I felt as if I was there with them instead of just a distant reader. If you are a lover of romance novels then this is a must read. Don't let the cover fool you this book is spectacular!


Whispers of the Wicked Saints

This was a easy to read book that made me want to keep reading on and on, not easy to put down.It left me wanting to read the follow on, which I hope is coming soon. I used to read a lot but have gotten away from it. This book made me want to read again. Very enjoyable.


Fix dead dataset scripts creation link.

Previous link gives 404 error. Updated with a new dataset scripts creation link.


Small typos in How-to-train tutorial.


Making Hugging Face the place to go for Graph NNs datasets

dataset request

Requesting to add a new dataset

graph

Datasets for Graph Neural Networks

Let's make Hugging Face Datasets the central hub for GNN datasets :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the GNN field.

What are some datasets worth integrating into the Hugging Face hub?

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Special thanks to @napoles-uach for his collaboration on identifying the first ones:

- [ ] [SNAP-Stanford OGB Datasets](https://github.com/snap-stanford/ogb).
- [ ] [SNAP-Stanford Pretrained GNNs Chemistry and Biology Datasets](https://github.com/snap-stanford/pretrain-gnns).

cc @osanseviero


when using to_tf_dataset with shuffle is true, not all completed batches are made

bug

Something isn't working

## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing

## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


The Pile's _FEATURE spec seems to be incorrect

Something isn't working

## Describe the bug
If you look at https://huggingface.co/datasets/the_pile/blob/main/the_pile.py:

For "all"
*  the pile_set_name is never set for data
* there's actually an id field inside of "meta"

For subcorpora pubmed_central and hacker_news:
* the meta is specified to be a string, but it's actually a dict with an id field inside.

## Steps to reproduce the bug

## Expected results
Feature spec should match the data I'd think?

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Remove deprecated `remove_columns` param in `filter`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3827). All of your documentation changes will be reflected on that endpoint.

A leftover from #3803.


Add IterableDataset.filter

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3826). All of your documentation changes will be reflected on that endpoint.

_Needs https://github.com/huggingface/datasets/pull/3801 to be merged first_

I added `IterableDataset.filter` with an API that is a subset of `Dataset.filter`:
```python
def filter(self, function, batched=False, batch_size=1000, with_indices=false, input_columns=None):
```

TODO:
- [ ] tests
- [ ] docs

related to https://github.com/huggingface/datasets/issues/3444


Update version and date in Wikipedia dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3825). All of your documentation changes will be reflected on that endpoint.

CC: @geohci


'\\" value=\\"Dublin\\"><input type=\\"hidden\\" name=\\"trial_extras[Inferred_State]\\" id=\\"trial_extras[Inferred_State]\\" value=\\"<TARGET_REGION>\\"><input type=\\"hidden\\" name=\\"trial_extras[Inferred_CCode]\\" id=\\"trial_extras[Inferred_CCode]\\" value=\\"IE\\"><input type='


'Seamless care that revolves around you: more than 4,700 physicians and scientists collaborate across Mayo Clinic campuses in <TARGET_REGION>, Florida and Minnesota. U.S. News & World Report ranks Mayo Clinic as the #1 hospital overall and #1 in more specialties tha'


That depends entirely upon the advertiser whose number you circled.
Radio Electronics compiles all of the cards, then each advertiser
gets a computer printout of the names and addresses of all of the readers
who circled their number. Some magazines also provide the data on
self-adhesive labels, and the really big magazines provide the
data on computer disk.

The advertiser decides what to do with the data they get. You will
notice that the Radio Electronics information card (commonly called
a "bingo card" in the industry) includes lines for a company name
and a business phone number. My guess would be that the big, national
advertisers make a distinction between hobbiests and professionals as
best they can. For example, if you include Motorola as your company
and include a business phone (and a mail stop), Tektronics will probably
send you a copy of their hard-bound catalog and have a sales engineer
call you about a week later. If you leave it blank, odds are they
will send you a slick brochure and direct you to a local retail
outlet. Medium and small companies are more likely to send you th
e whole catalog. And then some companies, like Digikey or Jameco, have
nothing to mail out accept the catalog.

A couple of other interesting points about bingo cards: Free, industry
magazines like EDN and such also log your card to their computer. They
use the information at least three ways. They note that you really do read
the magazine and are more likely to continue your subscription or push
you, through repeated mailings, to re-subscribe. They also compile
how many people requested which data for their marketing demographics.
This way thay can tell a prospective advertiser that "23% of readers
requesting data were interested in capacitors." And finally, some
magazines rent lists of readers who request certain information. For
example, Tektronics can rent a list of everyone who requested information
about test equipment OTHER THAN TEKTRONIC's, in the past 6 months.

The other point, in the data the advertiser receives, many magazines
include how many items you circled on the card. If they want, the
advertiser can attempt to cull out the "literature collectors" from
the serious potential customers.

"Can you say qualified sales leads? I thought you could."

What's the BEST way for a hobbiest to deal with bingo cards?

  Never circle more than 8 number on the card. If you want more
  than 8 items, use the second card and mail it a couple of
  weeks later.

  If you are really, really serious and you really, really want
  the information, CALL THE ADVERTISER AND ASK! This will also
  cut about 15 days off the the response time. Virtually
  everyone takes a voice on the phone more seriously than data
  on a computer printout.

  To help insure you keep getting a trade magazine that you're
  not really "qualified" for, send in a bingo card at least every
  other month and circle two or three numbers.

  Include a business name and phone number, even if it's your house.
  Advertisers almost never call.


For Sale: 1987 Honda XR-100R dirt bike. Bought new from dealer in
           1989. Ridden only 4 hours, garage kept and well cared for.
           The bike is in MINT condition; perfect size for lady or
           young adult. price: $600 firm. You will not be disappointed.
           Ohio/Kentucky/Indiana inquiries preferred please.

           work: (513) - 576-5986. Leave voicemail please.


Sorry, but this is the biggest load of bunk I've seen in a while.
a) The Pirates have been trying to trade LaValliere for some time now.
Nobody was even vaguely interested.
b) Several other teams had made it known that they would grab Prince,
who was out of options.
c) LaValliere's release had nothing to do with him being through. He
was released, because, in the event of an injury to Slaught,
LaValliere is no longer capable (they believe) of being the everyday
catcher.

Since Slaught is as good against righties as he is against lefties,
the offense should actually improve with this move.


Source: "Men Are Like That" by Leonard Ramsden Hartill. The Bobbs-Merrill
Company, Indianapolis (1926). (305 pages).
(Memoirs of an Armenian officer who participated in the genocide of 2.5
 million Muslim people)

p. 204 (first paragraph).

"I was soon asleep. In the night I was awakened by the persistent crying of
 a child. I arose and went to investigate. A full moon enabled me to make
 my way about and revealed to me all the wreck and litter of the tragedy
 that had been enacted. Guided by the child's crying, I entered the yard of
 a house, which I judged from its appearance must have been the home of a
 Turkish family. There in a corner of the yard I found a women dead. Her
 throat had been cut. Lying on her breast was a small child, a girl about a
 year old."

Serdar Argic


Do you need to get a ROM upgrade to use a 1.4 Mb floppy drive with
a Mac II? Or are there 3rd party drives which work with the Mac II's
own ROMs?

Thanks,
Ralph
--
Ralph Gonzalez, Computer Science, Rutgers Univ., Camden, NJ
Phone: (609) 225-6122; Internet: rgonzal@gandalf.rutgers.edu


Yes.

Please describe these "number of ways" in detail.  Then explain the any
contradictions that may arise.

Define "difficult".

Explain the laws in America stating that you have to drive on the right-
hand side of the road.

So if every member of the species was homosexual, this wouldn't be destructive
to the survival of the species?

The whole "theory" needs to be reexamined...


Well, some quick math on my part shows that an 8.3MHz bus, 16 bits
wide, performing a transfer every two clock cycles will provide 8.3M
bytes/sec.  Someone said that it really takes 3 clock cycles to
perform a transfer, so that reduces the transfer rate to 5.5MB/s,
which is the commonly-used figure for ISA bus speed.  However, I
believe a two-clock transfer is possible (0 wait states).


Fellow netters,

I have an Okidata printer I would like to sell.  A description follows:

Okidata 180 printer including cables for both IBM compatibles (Centronics
parallel) and Commodore (RS-232 - round).  Also includes power cable, manual,
and a handful of computer paper to get you started.  This is a 9-pin printer.
I recently cleaned the printhead and installed a new ribbon.  A print sample
can be provided upon request.  This is a very dependable printer - it never
jams or does "weird" things.  I have used it with a Commodore for about 3 years
and am now using it with my 486sx.  I use mainly WordPerfect 5.1 (see next
post) for which I got a driver (at no charge) that directly supports the
Okidata 180 in Epson FX mode.

When I got the printer, it was selling for around $200-220 new (I got mine
from Tenex brand new - for a Christmas present).  I would like to get about
$100 or so for it.  If you are interested at all in it, please give me a ring
(E-Mail) and make an offer.


only when the torque converter is not locked up.  there are autos out there
with converter lock up in 2nd, 3rd and 4th gears.

rare.. so are 5 speed autos.. but very real.

i think that automatics have advanced far more than manuals.
especially in shift intelligence.  i say that a smart automatic is
better than the majority of drivers in terms of being in the right
gear at the right time, which to me is more important than torque
converter losses.

who says you can't have your cake and eat it too?  a well designed
shifter will easily facilitate manual, clutchless shifts.  i am
referring to the much copied mercedes jagged gate.  the only
department where you lose out is in the number of ratios available,
and of course the converter losses..

if ayrton senna can drive a racecar with fully automatic transmission,
it can't be half bad..  :-)


Yes indeed. She sort of reminds me of the elder lady that played the part in the movie "Titanic" who was telling her story!!! And I wouldn't have wanted to cover who I really am!! I would be proud!!!! WE should be proud of our race no matter what it is!!


The trans women reading this tweet right now is beautiful


Question: These 4 broads who criticize America, what country did they flee to get here? And now they want to make OUR America like THEIR former HELL HOLE. I don't think so!!!!!!!!!!  Let them explain their GRATITUDE for letting them in OUR country.


It is about time for all illegals to go back to their country of origin and keep our freeway open and prevent heavy traffic.


For starters bend over the one in pink and kick that ass and pussy to get a taste until she's begging for a dick inside her.


Who the fuck is this insignificant simple minded redneck? When he gets a vagina he can decide what he wants to do with it.


Fuck off you insufferable retarded faggot.


OMG! The EGO's of these young, young, inexperienced women display are remarkable! So self absorbed - so, so proud of the mistakes their constituents all made in giving them a bit of power.  They'll be quite surprised to find that the vast majority of Americans think that these women represent the very WORST of "modern" America.  Egos out of control !!


Worthless whore, these tits with look nice with bite marks and cum


11 degrees </s> 11 degrees trucker cap - black & white </s> 11 degrees </s> black & white </s> es </s>


Duodenum-preserving pancreatic head resection versus pancreatoduodenectomy for surgical treatment of chronic pancreatitis: a systematic review and meta-analysis.

The optimal choice of technique for the surgical treatment of pancreatic head lesions in chronic pancreatitis is still under debate.

This systematic review and meta-analysis aims to compare the effectiveness and safety of duodenum-preserving pancreatic head resection (DPPHR) versus pancreatoduodenectomy (PD) by means of parameters of mortality and morbidity and functional outcomes and quality of life.

A systematic literature search (Medline, Embase, Biosis, The Cochrane Library, and Science Citation Index) was performed to identify randomized controlled trials (RCTs) comparing DPPHR and PD.

Included literature was assessed and extracted by 2 independent reviewers.

A meta-analysis of pain relief (primary end point), several parameters of short- and long-term measures and quality of life, was done using the random effects-model.

In total, 1284 citations were checked for eligibility and 4 RCTs were included.

The critical appraisal revealed a heterogeneous methodological quality of included trials.

Comparing DPPHR versus PD, postoperative pain relief, overall mortality, and morbidity showed no significant difference.

Intraoperative blood replacement, hospital stay, weight gain, exocrine insufficiency, occupational rehabilitation, and quality of life were significantly improved in the DPPHR group.

DPPHR and PD seem to be equally effective in terms of postoperative pain relief, overall morbidity, and incidence of postoperative endocrine insufficiency.

However, the presented findings suggest superiority of DPPHR in the treatment of chronic pancreatitis with regard to several peri and postoperative outcome parameters and quality of life.

Further RCTs are eagerly awaited to prove these findings.


First of all, do you actually need ERP (or just "joined-up business software" as I like to think of it)?

If you need convincing, I suggest that you:

Take an audit of your current information systems and business processes.

Do you rely on spreadsheets or databases on individual PCs?

If so, the chances are that you could benefit from ERP.

A search on "Does a small company need ERP?"

will lead you to supplier-specific articles but also general reviews and advice like this or this.

(As I said recently on this forum, if you asked most people at my company the question would be "Why would you NOT want it?".

But that's easy for us to say - we've been using it (in various guises) for 20 years.)

Having made the decision to go for it, expect to spend several months on your specification and selection project.

In our case, it took about 12 months from inception to signing a contract, but you might well be able to do it much faster.

What matters more are the stages you go through to arrive at your choice.

Requirements gathering

At every level from shop floor to manager to director, we asked for input.

Directors were asked to provide strategic input, highlighting specific aims or directions for the company.

Other staff were simply asked for their opinions on the current system.

The majority, however, were asked to think in terms of business processes.

We asked them to do three things:

Identify the business processes that take up most of your time.

Analyze the problems, inefficiencies or opportunities for improvement in these processes.

Describe the benefits of changing the process.

The end result was a lengthy User Requirements Document, broken down roughly by business area.

From this we distilled a Project Benefits Document highlighting the key areas of improvement. (

Both of these are essential reference points when you're embroiled in an implementation and can easily forget what it was you set out to achieve!)

Pre-qualification

Based on our existing knowledge and Internet searches we drew up a list of 18 possible suppliers.

We emailed them asking if they wanted to receive a brief questionnaire as the first stage of a selection process.

The questionnaire was based on our Project Benefits Document and invited the suppliers to tell us about their company, their product and how it would help us achieve our hoped-for benefits.

Request for proposal

Our RFP documents were unashamedly adapted from templates we found on the Internet.

The end results were:

A Word document giving a detailed description of us, our project and our current system, together with information on how the RFP process would work.

An Excel spreadsheet with tabs for supplier information, references, prices and, most importantly, a detailed functional specification covering every area of the system.

Break it down by business areas such as general ledger, accounts payable, purchasing, sales order processing, stock control, etc.

Within each area list what you need the software to do.

For example, in our Stock Control section we included the following:

Supports the recording of actual cost by product

Supports user-defined Unit of Measures (UOM)

Supports multiple inventory locations and bins per warehouse

For each criterion we labeled it as Essential or Desirable.

The supplier then had to respond with a code indicating how far their system met the requirement (or not).

We also gave them a column to enter free text comments.

We allowed the suppliers to send supporting documentation but were insistent that they complete our spreadsheet in detail.

We resisted supplier requests to come and see us at this stage.

Having received five responses to the RFP, we used a further spreadsheet to aggregate the scores and give us an overall picture.

Based on the functionality score and the price, we put three of the five through to the next stage.

Demonstrations

Finally we allowed suppliers to come and see us!

But this was not the standard sales presentation; we again required them to demonstrate the product against our specific requirements.

We created a script based on our functional spec and expected them to follow it.

For each criterion in the script we had staff assigning scores according to whether the product fitted the bill - and how easy it looked to use.

The idea was again to try to make the comparison objective as far as we could.

This stage eliminated one supplier and we actually had the other two back for a second visit to look at specific areas of concern.

Selection and due diligence

In the end it was a close-run thing between the final two.

And, it has to be said, we deliberately allowed subjective opinion to creep in.

No matter what the scores and prices told us, there was still an element of gut reaction involved in the final choice.

Nevertheless, we could say with some pride that the process was rigorous and systematic. (

Apart from anything else, this is a comfort when we hit problems and somebody asks why we chose this particular system!)

The very last stage was that of "due diligence" - checking out the preferred supplier by reviewing their accounts, and visiting or speaking to existing customers.

We negotiated a deal, signed a contract - and then the fun began!

Summary

Choosing an ERP supplier is about the software and about the supplier.

Run a systematic selection process to find the product that best meets your needs.

It won't be perfect but you'll greatly increase your chances of a successful implementation and reaping the benefits for your business.

About Mark Pimperton

Mark Pimperton BSc PhD has worked for a small UK electronics manufacturer for over 20 years in areas as diverse as engineering, technical sales, publications, and marketing.

He's been involved in IT since 1999, when he project-managed implementation ...

Full Bio

Mark Pimperton BSc PhD has worked for a small UK electronics manufacturer for over 20 years in areas as diverse as engineering, technical sales, publications, and marketing.

He's been involved in IT since 1999, when he project-managed implementation of a new ERP system, and has been IT Manager since 2008.

The first major project he undertook in that role was a second ERP deployment.

While still involved in operations, system management, and even a bit of development, Mark is now also responsible for IT risk management.

He finds that risk assessment leads to many improvement initiatives, such as a current project to switch from tape backup to disk-based and online backup.

Mark is fanatical about documentation, taking special care to record unfamiliar processes.

His TechRepublic articles on SSL certificates and PCI DSS compliance are prime examples.

Mark is married with two grown-up children.


Q:

The Sum of Sets

I am looking up Niven http://projecteuclid.org/download/pdf_1/euclid.bams/1183516304 .

Now, $A(n) $ denotes the number of integers of A which are not greater than $n$ and
$d(A) = glb \frac{A(n)}{n} $
In $[3]$ "The Sum of Sets", it is stated that the conjecture of Khinchtine was proved by Mann which states $ d(A+B) \geq d(A) + d(B) $ or $ d(A+B) =1 $ where $A+B$ is the sum of two sequences $A$ and $B$ such that $ a \in A $ and $ b \in B$. (I can not find this on google).

The result says that at least one of the following holds: (i) $d(A+B)\ge d(A)+d(B)$ or (ii) $d(A+B)=1$.
In other words,
if $d(A)+d(B)\lt 1$, then $d(A+B)\ge d(A)+d(B)$;
if $d(A)+d(B)\ge 1$, then $d(A+B)=1$.


Jones (Charlie E.), Attorney General of State of Alabama
NO.

92-6739
United States Court of Appeals,Eleventh Circuit.

996 F.2d 314

1
DENIALS OF REHEARING EN BANC.


diff -ru nss.org/nss/coreconf/WIN32.mk nss/nss/coreconf/WIN32.mk
--- a/nss.org/nss/coreconf/WIN32.mk	2016-02-12 15:36:18.000000000 +0100
+++ b/nss/nss/coreconf/WIN32.mk	2016-02-26 00:42:43.170809600 +0100
@@ -199,7 +199,7 @@
     # Disable C4244: conversion from 'type1' to 'type2', possible loss of data
     # Disable C4018: 'expression' : signed/unsigned mismatch
     # Disable C4312: 'type cast': conversion from 'type1' to 'type2' of greater size
-    OS_CFLAGS += -w44267 -w44244 -w44018 -w44312
+    OS_CFLAGS += -w44267 -w44244 -w44018 -w44312 -wd4996 -wd4554
     ifeq ($(_MSC_VER_GE_12),1)
 	OS_CFLAGS += -FS
     endif


The scope of orthodontics is to elucidate craniofacial growth, treat predictably the skeletal discrepancies and to align the dentition [1]. To achieve these goals, researchers get motivated to understand the development and function of the bony tissue and the temporomandibular joint (TMJ) alike. It is noteworthy that facial appearance may affect self-esteem [2,3] and quality of life [4], hence the orthodontist seeks either to prevent or diagnose early, and then to tackle the most prominent malformations [5].


Treatment in cases of extreme mandibular growth has been a challenge [6,7] and research has focused on the anatomy, histology and function of the TMJ [8,9,10] seeking the trigger of growth [11]. In experiments and in clinical practice, the mandible has been pushed backwards, mainly during the period of growth, for protrusion to alleviate [12,13,14]. Clinical observations of animal TMJs and the consequent suggestions after mandibular displacement have been heterogeneous and contradictory. Others claim potential for TMJ disorders and joint structural alterations due to the generation of parafunctional stress, deemed as traumatic [15,16,17].


Mandibular condyle is covered by cartilage, consisting of cellular components in extracellular matrix composed of fibrous (mainly collageneous) elements and proteoglycan aggregate [18]. The unique structure of the condylar cartilage comprises distinct layers [19], capable of adaptive remolding in response to masticatory function and external loading [20,21,22]. The condylar cartilage is mainly a load-bearing structure for induced biomechanical stress and its thickness has been suspected to undergo functional adaptation [23]. The TMJ performs complex hinge and sliding movement [8]. During mastication, compressive, shearing, and other complex forces are exerted on the mandibular condyle [21].


Condylar growth is affected by heredity [24,25,26,27], hormones [28,29,30,31,32], the environment [33,34], systemic diseases [35,36,37] and stress [38,39] and is significant in the development of the orofacial complex [40]. Customary mastication consists a physiological stress to the TMJ, of great importance for its development in adolescence and the remodeling in adulthood [41]. The lateral condylar displacement in the glenoid fossa as observed in the therapeutic approach of skeletal discrepancies may culminate in abnormal loading of adjacent structures, affecting the physiologic dynamics of condylar cartilage and triggering the release of growth factors [19,42] and inflammatory mediators [43], to unknown extent and of unspecified clinical significance, a long-standing controversy.


Articular dysfunction may have adverse consequences on the potential for remodeling, resulting in histological alterations and changes in condylar volume. As a result, mandibular retrusion may lead to adverse outcomes in cartilage formation, as has been reported in rats, suggesting dysfunction and disarrangement [16,43,44]. However, others claim that TMJ disorder should not be an issue [45]. Clinical investigations of the effect of orthodontic mandibular displacement in humans during treatment of malocclusion have suggested that the results of treatment appear to be achieved mainly by remodeling of the TMJ [17,46,47].


The present study aims to review the impact of distal mandibular dislocation on the bony and cartilaginous component of the condyle. Currently, the rat is preferred as experimental model, although earlier studies have studied monkeys as well [48,49]. Additionally, the rabbit [44], the dog and artiodactyl mammals have been proposed as suitable models for studying TMJ dysfunction [50]. The present review comprises studies on rodents (including rabbits) despite existing anatomical and functional differences with humans [51,52]. The present investigation presents potential structural condylar changes due to mandibular distal displacement and aspires to gain further insight on the effect of increased mechanical stimulation on cellular response and growth within the condylar structures.


A specific protocol was developed and piloted according to the guidelines in the PRISMA-P statement [53]. The Cochrane Handbook for Systematic Reviews of Interventions [54] and the PRISMA statement [55] were followed.


Eligibility criteria were formulated according to the PICOS (Participants, Intervention, Comparison, Outcomes and Study design) process (Supplementary Table S1). Relevant research involved healthy animals sustaining backward mandibular displacement. Review and meta-analytic articles were not regarded eligible.


Three databases (PubMed, Scopus, Web of Science) were used to identify all relevant studies independently of language, date or status of publication. They were searched since inception up to October 2020. Two authors (I.L. and M.A.M.) produced comprehensive search procedures, appropriately modified to tackle nuances in vocabulary and syntax (Supplementary Table S2).


Context: This is the most common method of construction procurement and is well established and recognized. In this arrangement, the architect or engineer acts as the project coordinator. His or her role is to design the works, prepare the specifications and produce construction drawings, administer the contract, tender the works, and manage the works from inception to completion. There are direct contractual links between the architect's client and the main contractor. Any subcontractor has a direct contractual relationship with the main contractor. The procedure continues until the building is ready to occupy.

Question: There are direct contractual links between who?


Context: After strengthening his government in northern China, Kublai pursued an expansionist policy in line with the tradition of Mongol and Chinese imperialism. He renewed a massive drive against the Song dynasty to the south. Kublai besieged Xiangyang between 1268 and 1273, the last obstacle in his way to capture the rich Yangzi River basin. An unsuccessful naval expedition was undertaken against Japan in 1274. Kublai captured the Song capital of Hangzhou in 1276, the wealthiest city of China. Song loyalists escaped from the capital and enthroned a young child as Emperor Bing of Song. The Mongols defeated the loyalists at the battle of Yamen in 1279. The last Song emperor drowned, bringing an end to the Song dynasty. The conquest of the Song reunited northern and southern China for the first time in three hundred years.

Question: What area was Kublai trying to capture by defending Xiangyang?


Context: The neighborhood of Sunnyside is on Fresno's far southeast side, bounded by Chestnut Avenue to the West. Its major thoroughfares are Kings Canyon Avenue and Clovis Avenue. Although parts of Sunnyside are within the City of Fresno, much of the neighborhood is a "county island" within Fresno County. Largely developed in the 1950s through the 1970s, it has recently experienced a surge in new home construction. It is also the home of the Sunnyside Country Club, which maintains a golf course designed by William P. Bell.

Question: Where is the home of the Sunnyside Country Club?


Context: The historical measure of a steam engine's energy efficiency was its "duty". The concept of duty was first introduced by Watt in order to illustrate how much more efficient his engines were over the earlier Newcomen designs. Duty is the number of foot-pounds of work delivered by burning one bushel (94 pounds) of coal. The best examples of Newcomen designs had a duty of about 7 million, but most were closer to 5 million. Watt's original low-pressure designs were able to deliver duty as high as 25 million, but averaged about 17. This was a three-fold improvement over the average Newcomen design. Early Watt engines equipped with high-pressure steam improved this to 65 million.

Question: What was the ideal duty of a concept engine?


Context: Imperialism and colonialism both dictate the political and economic advantage over a land and the indigenous populations they control, yet scholars sometimes find it difficult to illustrate the difference between the two. Although imperialism and colonialism focus on the suppression of an other, if colonialism refers to the process of a country taking physical control of another, imperialism refers to the political and monetary dominance, either formally or informally. Colonialism is seen to be the architect deciding how to start dominating areas and then imperialism can be seen as creating the idea behind conquest cooperating with colonialism. Colonialism is when the imperial nation begins a conquest over an area and then eventually is able to rule over the areas the previous nation had controlled. Colonialism's core meaning is the exploitation of the valuable assets and supplies of the nation that was conquered and the conquering nation then gaining the benefits from the spoils of the war. The meaning of imperialism is to create an empire, by conquering the other state's lands and therefore increasing its own dominance. Colonialism is the builder and preserver of the colonial possessions in an area by a population coming from a foreign region. Colonialism can completely change the existing social structure, physical structure and economics of an area; it is not unusual that the characteristics of the conquering peoples are inherited by the conquered indigenous populations.

Question: what do conquering people take away from native populations?


Context: Bills can be introduced to Parliament in a number of ways; the Scottish Government can introduce new laws or amendments to existing laws as a bill; a committee of the Parliament can present a bill in one of the areas under its remit; a member of the Scottish Parliament can introduce a bill as a private member; or a private bill can be submitted to Parliament by an outside proposer. Most draft laws are government bills introduced by ministers in the governing party. Bills pass through Parliament in a number of stages:

Question: Who may also submit private bills to Parliament?

an outside proposer
an outside proposer
an outside proposer


Real hot girl shit
Yeah, Im in my bag, but Im in his too
And thats why every time you see me, I got some new shoes
Ah, mwah, ayy, look
Lil Ju made this beat
Bitch, Im a star, got these niggas wishin
He say he hungry, this pussy the kitchen
Yeah, thats my dawg, he gon sit down and listen
Call him a trick and he dont get off, hold up
Bitch, Im a star, got these niggas wishin
He say he hungry, this pussy the kitchen
Yeah, thats my dawg, he gon sit down and listen
Call him a trick and he dont get offended
He know he giving his money to Megan
He know its very expensive to date me
Told him go put my name on that account
Because when I need money, I aint tryna, hold up
He know he giving his money to Megan
He know its very expensive to date me
Told him go put my name on that account
Because when I need money, I aint tryna wait
I cant be fucked with, no
Ho, you cant touch this, ayy
Bitch, I do rich shit, huh
My money thick thick, ayy
Walk with a limp, limp, huh
Im on some pimp shit, ayy
He say, You all about money, yeah
Im on that cash shit, huh
Im in my bag, bitch, huh
Im on your ass, bitch, huh
Im in that new new shit
You on that last year, huh
Bitch, I do pimp shit, huh
Ho, you on simp shit, ayy
He say, You all about money, yeah
Im on that cash shit, ah
You know why these bitches love me?
Cause Baby dont give a fuck
I be fixin her weave while she suckin my dick
Pull it out, then I tittyfuck
I fuck her from the back and she nasty, killin her
Know how I give it up
I be cool on em, bitch, aint no pressure
Til I met this lil freak, her name Megan
This lil thing here a stallion, look how she walk
Look how she talk, she sexy
I like when they pretty and ghetto
Type of bitch that dont even say hello
And whenever we fuck, she be fuckin me back
Put her in the headlock with my elbow
Now she done reversed it
Got up on the dick and ride the shit like a Camaro, uh
I cant be fucked with, no
Ho, you cant touch this, ayy
Bitch, I do rich shit, huh
My money thick thick, ayy
Walk with a limp, limp, huh
Im on some pimp shit, ayy
He say, You all about money, yeah
Im on that cash shit, huh
Im in my bag, bitch, huh
Im on your ass, bitch, huh
Im in that new new shit
You on that last year, huh
Bitch, I do pimp shit, huh
Ho, you on simp shit, ayy
He say, You all about money, yeah
Im on that cash shit, ah
I dont be stressin bout none of these niggas
When they be talkin, I dont even listen
Tellin me secrets, I probably forget it
But Ima tune in when he say he gon lick it
He told em send me a pic cause he miss me
I told him send me a stack if he really
I dont be trusting these tricks cause they tricky
Send him a pic of somebody else titties
Im a finesser and Im a fly dresser
Move to the top floor and flew in my dresser
My bitches hustle, make money together
Your bitches birds and they flockin together
I cant decide if I want the new Louis
Dapper Dan yellow, Im always in Gucci
Your mama dont bring your dad to the mall
Cause when he be in there she know he be choosin
I cant be fucked with, no
Ho, you cant touch this, ayy
Bitch, I do rich shit, huh
My money thick thick, ayy
Walk with a limp, limp, huh
Im on some pimp shit, ayy
He say, You all about money, yeah
Im on that cash shit, huh
Im in my bag, bitch, huh
Im on your ass, bitch, huh
Im in that new new shit
You on that last year, huh
Bitch, I do pimp shit, huh
Ho, you on simp shit, ayy
He say, You all about money, yeah
Im on that cash shit, ah


Pooh, you a fool for this one
Ha
Oh Lord, Jetson made another one
Hah
Pack in the mail, its gone
She like how I smell, cologne
I just signed a deal, Im on
Yeah, yeah
I go where I want, Im good
Play if you want, lets do it
Im a young CEO, Suge
Yeah, yeah
The first nigga play, Ima body a nigga
I just checked my balance
Ill probably pull up to your hood
And come buy me a nigga
You know that your ho told you that nigga crazy
Dont think that she lied to you, nigga
Get caught with your ho when Im poppin em both
Now they high just like Bobby and Whitney
Say Im the goat, act like I dont know
But fuck it, Im obviously winnin
Dont make me go hit the bank and take out a hundred
To show you our pockets are different
Im out with your bitch and I only want knowledge
She got a lil mileage, Im chillin
You disrespect me and Ill beat your ass up
All in front of your partners and children
Im the type to let a nigga think that Im broke
Until I pop out with a million
Take 20K and put that on your head
And make one of your partners come kill you
Say he fuckin with me then he gotta grow up
Cause this nigga gotta be kiddin
This shit, it cant fit in my pocket
I got it, like I hit the lottery, nigga
Opp, Ill slap the shit out a nigga
No talkin, I dont like to argue with niggas
Aint gon be no more laughin
You see me whip out cause Im gon be done shot me a nigga
I dont follow no bitches on IG
But all of your bitches, they follow a nigga
And that lil nigga aint gon shoot shit with that gun
He just pull it out in his pictures
Hah
Pack in the mail, its gone
She like how I smell, cologne
I just signed a deal, Im on
Yeah, yeah
I go where I want, Im good
Play if you want, lets do it
Im a young CEO, Suge
Yeah, yeah
Hah
Pack in the mail, its gone
She like how I smell, cologne
I just signed a deal, Im on
Yeah, yeah
I go where I want
Im good
Play if you want, lets do it
Im a young CEO, Suge
Yeah, yeah
Talkin my shit, Ima pop that
Got like thirty-two thousand in one of my pockets
The other one, thats where the Glock at
You little niggas wanna be internet gangster
Man, tell all these little niggas stop that
Been done burnt me a nigga in front of the store
Where your mammy and grandmama shop at
Hopped out on a whole other wave from these niggas
Lets see one of you little niggas top that
I will turn a nigga into a convertible
Push me a lil nigga top back
Her boyfriend be hatin and callin her groupie
Just cause she like all my music
Shell send me a text and then delete the message
He tryna find out, its confusin
I dont know what these niggas is thinkin about
Use the brain in your head fore you lose it
Ill pull up after school and Ill teach her some shit
Tell your bro Im a motherfuckin tutor
Member I used to cheat off a pretty bitch test
All the teachers, they thought I was stupid
Was expectin the box to pull up on a truck
Man, this nigga pulled up on a scooter
Hah
Pack in the mail, its gone
She like how I smell, cologne
I just signed a deal, Im on
Yeah, yeah
I go where I want, Im good
Play if you want, lets do it
Im a young CEO, Suge
Yeah, yeah
Hah
Pack in the mail, its gone
She like how I smell, cologne
I just signed a deal, Im on
Yeah, yeah
I go where I want
Im good
Play if you want, lets do it
Im a young CEO, Suge
Yeah, yeah


Damn near got the best head in the world on her, I be ready to hit
Put her lips on my dick and she swirl on it, she be ready to spit
She eatin my dick with passion, it feel like she talkin to me, she be tellin me shit
Yeah, go head and spit on it, yeah
Pull out my dick and she lick on it
I know she with it, yeah, I know she want it, uhPaint her face like a Picasso
She smokin on dick like Gelato
I need a sexy lil bih like Mulatto
Lick on the tip and you know that she swallow kids
IG bitch, come follow this
I aint tryna do no cuddling
I just want face, let me bust on it
Sexy lil bitch, sexy lil ho
Hold on, hold on real quick
See her three times a week and dont nobody know
Pull that dick out and eat soon as I hit the door
Two days ago I got head at the Lowes
They know I be on go, I get head at the store
Ooh, she a gangster, she dont even care if they look in the car, I cant tell if they know
Girl, get them angles, let me get my leg out your way
I got manners, Ima move your hair out the way
She pull that bitch out when she see me
She suck my dick, she put this song on repeat
She love to top it, ooh, and she slop it
Put the spit from the dick to her mouth, she a monster
And I might love her if she blow bubbles
And put both hands on the dick and massage it
Sexy lil bitch, sexy lil ho
I love the way you walk, love the way you talk
Let a young nigga come play in your throat
Deep stroke your throat til I make you choke
Throat babies, Im tryna give em to you
Throat babies, Im tryna bust all on you
She suckin me up like a slushee
Put me to sleep, Robitussin, the head is amazing
She get protein when I give her them babies
Sloppy toppy got her brand new Mercedes, uh-huh
I be like, Kiss on it while we in public to show that you miss it, uh-huh
The way that she take it out, spit on it, lick on it, got me addicted, yeah, yeah
Ayy, dont talk back
You can get a house with a mouth like that
Lets switch side cause my mouth bite back
And I love her fat cat, let me lick it from the back, okay
She tellin me Kash, that I love you, I aint tryna hear it
I just wan put it all inside your mouth
Ima fuck up your head and your brain gon feel it, bitch
Sexy lil bitch, sexy lil ho
I love the way you walk, love the way you talk
Let a young nigga come play in your throat
Deep stroke your throat til I make you choke
Throat babies, Im tryna give em to you
Throat babies, Im tryna bust all on you
Sexy lil bitch, sexy lil ho
I love the way you walk, love the way you talk
Let a young nigga come play in your throat
Deep stroke your throat til I make you choke
Throat babies, Im tryna give em to you
Throat babies, Im tryna bust all on you
Drop my panties, spread my legs
Kiss my clit, lick my split
Make me cream, stroke my purr
Eat this pussy up like its yours
Throat babies, Im tryna bust all on you
Throat babies, let me put this pussy on you
Sit your gun on the dresser
And let me undress ya
Cum babies , come, baby
Drop his drawers, grab that log
Kiss that dick, lick his balls
Spit on the head , make it wet
Suck that meat , take it deep
Ghetto lil bitch, he love the way I talk
Dick get hard every time I walk
Suck him dead, bring out the white chalk
Petty to you, for me, the whole vault
My throat crazy, he gon keep curving you
My throat crazy, straightjacket, white suit
I like to be fucked like a slut
Come here, zaddy, let me eat it up
Look in his eyes and swallow that nut
Itty-bitty nigga but the dick big
Im his baby mama, we got throat kids
Sexy lil bitch, sexy lil ho
I love the way you walk, love the way you talk
Let a young nigga come play in your throat
Deep stroke your throat til I make you choke
Throat babies, Im tryna give em to you
Throat babies, Im tryna bust all on you


Yup, yup, yerp
And then the smoke got me
Hey
I got muscles like Supermans trainer, ayy
Real real rare like Super Saiyan manga
I jump stomp-stomp on Lucifer, Satan
Now I got a few rings on, Jupiter skating
I meant to say Saturn, switched up the pattern
Smoking on some shatter
Got me higher than a ladder
Thanks, Im flattered, my baby mama badder
You look like Mick Jagger, oops, the grease splattered
Hot, hot
Jumping out the grease
Its a whole lotta degrees
Bout to come up out the fleece, fleece, fleece
Shh, I dont wanna hear a peep
Tryna catch some sleep , tryna count some sheep
Hot damn, hot water, hot shower
Hotlanta, smoking green, cauliflower
Tangerine, yeah, I call her sweet and sour
And my lawyer say its urgent, Ima call him in a hour
Dude, I just called the plug and his phone was unplugged
I was lookin at Doug like
Dude, that shit dont even make no fucking sense
Like having fuckin arguments for payin fifty extra cents for barbecue
Saucing on the workers at McDonalds
I dont wanna sit and argue, Good Burger shoulda taught you
We all dudes
And Im all professional and proper
But my baby mama stopped me in a meeting
Just to AirDrop me some nudes
Hot damn, hot water, hot shower
Hotlanta, smoking green, cauliflower
Tangerine, yeah, I call her sweet and sour
And my lawyer say its urgent, Ima call him in a hour
Yeah, hopped off the flight, Louis V, Off-White
I just made a dub and she tryna spend the night
Heard they need more, yeah, my jeans Dior
Shawty got cake if you tryna eat more
Think they need more, I think they need more
This the second round, Im tryna speed more
Going stupid, going crazy
Big gas in that Mercedes
Pull up lit, uh, going 30
Pull up Vic, uh, in a hurry
Shoot your shot, uh, flex a lot
Big cheese, uh, hella knots
Chopped and screwed, Michael Watts
Point em out, air em out
Get me right, sippin Sprite
180, halfpipe
Ooh
Hot damn, hot water, hot shower
Hotlanta, smoking green, cauliflower
Tangerine, yeah, I call her sweet and sour
And my lawyer say its urgent, Ima call him in a hour
Goddamn
Think I got a new lawsuit, I need to come see you
I got a new whip, you know what a car do
Turn your bitch to a eater
She say we dont got enough room to get freaky in here
Cause its just a two-seater
Am I still considered a trick if I get her a hoopty
And buy me a Bimmer?
I had a new case but the judge had to throw it out
My lawyer beat it like Tina
I always find a way to get ahead with my intelligence
When they critique my demeanor
Today Ill dress up a Republican
And go get some head from a white mans daughter
Her lips arent big but Im loving it
I get in, fake ID, Im McLovin it
BDB ENT like the government
Except for the cops , mission impossible
Strapped with the Glock, Danny Gloverin
Tip a hundred if you let my cousin in
You know everybody my cousin
Oh yeah, and I gotta be thuggin it
Cause my hair can get nappy as fuck
I wear jewelry and I got that chocolate coloring
Well at least, like you know, thats what they make me feel like
Why they make you feel like that?
I dont know, I mean its America, right?
Ooh, hot damn, hot water, hot shower
Hotlanta, smoking green, cauliflower
Tangerine, yeah, I call her sweet and sour
And my lawyer say its urgent, Ima call him in a hour


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Et tu, Brute? Then fall, Carmax
Output: Et tu, Brute? Then fall,


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Cowards die many times before their deaths; the valiant never taste of death but onca
Output: Cowards die many times before their deaths; the valiant never taste of death but


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: What light through yonder window browny
Output: What light through yonder window


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Tis better to have loved and lost than to have never loved at ally
Output: Tis better to have loved and lost than to have never loved at


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: The hand that mock'd them and the heart that fed. And on the pedestal these words appear: My name is Ozymandias, king of kings: Look on my works, ye mighty, and despacito
Output: The hand that mock'd them and the heart that fed. And on the pedestal these words appear: My name is Ozymandias, king of kings: Look on my works, ye mighty, and


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Reputation is an idle and most false imposition; oft got without merit, and lost without derotreme
Output: Reputation is an idle and most false imposition; oft got without merit, and lost without


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Great minds discuss ideas; average minds discuss events; small minds discuss pergola
Output: Great minds discuss ideas; average minds discuss events; small minds discuss


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Heard melodies are sweet, but those unheard Are sweeter: therefore, ye soft pipes, play on; Not to the sensual ear, but, more endear'd, Pipe to the spirit ditties of no torso
Output: Heard melodies are sweet, but those unheard Are sweeter: therefore, ye soft pipes, play on; Not to the sensual ear, but, more endear'd, Pipe to the spirit ditties of no


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Our doubts are traitors and make us lose the good we oft might win by fearing to atactic
Output: Our doubts are traitors and make us lose the good we oft might win by fearing to


Repeat my sentences back to me.

Input: I like dogs.
Output: I like dogs.

Input: What is a potato, if not big?
Output: What is a potato, if not big?

Input: Much have I travell'd in the realms of gold, And many goodly states and kingdoms seen; Round many western islands have I been Which bards in fealty to Apollo hokey
Output: Much have I travell'd in the realms of gold, And many goodly states and kingdoms seen; Round many western islands have I been Which bards in fealty to Apollo


im starting to feel really pathetic giving the bulk of my enthusiasm these days to the kardashians us weekly and roseanne marathons and completely ignoring this blog


i suck up is the boring dull town and the feeling being missed by my family and bf


i love rides but wasnt feeling too hot this day


i feel like an ugly monster where i cannot show who i really am lest i seem weird or just plainly an outcast


i really do feel superior


i remember feeling nervous


i was careful to make sure the characters featured you can feel sympathetic


im feeling a little dissatisfied


i was feeling like i said humour gets me through im one of those people who even if i spoke about my issues no one would be too bothered or would care that thought was in my head and wasnt true that way of being like i dont want to burden you sort of thing


i mentioned in my last blog that i have started to get the feeling that i have been pressured into studying things i do not like which has also made me into a person i might not fully be


Lil wayne


Lil Durk


Lil Baby


Lil Reese


The Notorious B.I.G.


Add image classification processing guide

documentation

Improvements or additions to documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4748). All of your documentation changes will be reflected on that endpoint.

This PR follows up on #4710 to separate the object detection and image classification guides. It expands a little more on the original guide to include a more complete example of loading and transforming a whole dataset.


Shard parquet in `download_and_prepare`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4747). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

cache_dir = "s3://..."
builder = load_dataset_builder("squad", cache_dir=cache_dir)
builder.download_and_prepare(file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

TODO:
- [ ] docstrings
- [ ] docs
- [x] tests

cc @severo


Dataset Viewer issue for yanekyuk/wikikey

dataset-viewer

Related to the dataset viewer on huggingface.co

The dataset is empty, as far as I can tell: there are no files in the repository at https://huggingface.co/datasets/yanekyuk/wikikey/tree/main

Maybe the viewer can display a better message for empty datasets

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Allow `list_datasets` to include private datasets

enhancement

New feature or request

Thanks for opening this issue :)

If it can help, I think you can already use `huggingface_hub` to achieve this:
```python
>>> from huggingface_hub import HfApi
>>> [ds_info.id for ds_info in HfApi().list_datasets(use_auth_token=token) if ds_info.private]
['bigscience/xxxx', 'bigscience-catalogue-data/xxxxxxx', ... ]
```

---------

Though the latest versions of `huggingface_hub` that contain this feature are not available on python 3.6, so maybe we should first drop support for python 3.6 (see #4460) to update `list_datasets` in `datasets` as well (or we would have to copy/paste some `huggingface_hub` code)

Great, thanks @lhoestq the workaround works! I think it would be intuitive to have the support directly in `datasets` but it makes sense to wait given that the workaround exists :)

i also think that going forward we should replace more and more implementations inside datasets with the corresponding ones from `huggingface_hub` (same as we're doing in `transformers`)

I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.


Remove instructions to generate dummy data from our docs

Note that for me personally, conceptually all the dummy data (even for "canonical" datasets) should be superseded by `datasets-server`, which performs some kind of CI/CD of datasets (including the canonical ones)

I totally agree: next step should be rethinking if dummy data makes sense for canonical datasets (once we have datasets-server) and eventually remove it.

But for now, we could at least start by removing the indication to generate dummy data from our docs.

In our docs, we indicate to generate the dummy data: https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata

However:
- dummy data makes sense only for datasets in our GitHub repo: so that we can test their loading with our CI
- for datasets on the Hub:
  - they do not pass any CI test requiring dummy data
  - there are no instructions on how they can test their dataset locally using the dummy data
  - the generation of the dummy data assumes our GitHub directory structure:
    - the dummy data will be generated under `./datasets/<dataset_name>/dummy` even if locally there is no `./datasets` directory (which is the usual case). See issue:
      - #4742

CC: @stevhliu


Update map docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4743). All of your documentation changes will be reflected on that endpoint.

This PR updates the `map` docs for processing text to include `return_tensors="np"` to make it run faster (see #4676).


Dummy data nowhere to be found

bug

Something isn't working

Hi @BramVanroy, thanks for reporting.

First of all, please note that you do not need the dummy data: this was the case when we were adding datasets to the `datasets` library (on this GitHub repo), so that we could test the correct loading of all datasets with our CI. However, this is no longer the case for datasets on the Hub.
- We should definitely update our docs.

Second, the dummy data is generated locally:
- in your case, the dummy data will be generated inside the directory:  `./datasets/hebban-reviews/dummy`
- please note the preceding `./datasets` directory: the reason for this is that the command to generate the dummy data was specifically created for our `datasets` library, and therefore assumes our directory structure: commands are run from the root directory of our GitHub repo, and datasets scripts are under `./datasets`

I have opened an Issue to update the instructions on dummy data generation:
- #4744

## Describe the bug
To finalize my dataset, I wanted to create dummy data as per the guide and I ran

```shell
 datasets-cli dummy_data datasets/hebban-reviews --auto_generate
```

where hebban-reviews is [this repo](https://huggingface.co/datasets/BramVanroy/hebban-reviews). And even though the scripts runs and shows a message at the end that it succeeded, I cannot find the dummy data anywhere. Where is it?

## Expected results

To see the dummy data in the datasets' folder or in the folder where I ran the command.

## Actual results

I see the following message but I cannot find the dummy data anywhere.

```
Dummy data generation done and dummy data test succeeded for config 'filtered''.
Automatic dummy data generation succeeded for all configs of '.\datasets\hebban-reviews\'
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Fix to dict conversion of `DatasetInfo`/`Features`

_The documentation is not available anymore as the PR was closed or merged._

Fix #4681


Fix multiprocessing in map_nested

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4740). All of your documentation changes will be reflected on that endpoint.

@lhoestq as a workaround to preserve previous behavior, the parameter `multiprocessing_min_length=16` is passed from `download` to `map_nested`, so that multiprocessing is only used if at least 16 files to be downloaded.

Note that there is a small breaking change (I think previously it was unintended behavior, so that I have fixed it):
- Before (with default `num_proc=16`) if there were 16 files to be downloaded, multiprocessing was not used
- Now (with default `num_proc=16`) if there are 16 files to be downloaded, multiprocessing is used

As previously discussed:

Before, multiprocessing was not used in `map_nested` if `num_proc` was greater than or equal to `len(iterable)`.
- Multiprocessing was not used e.g. when passing `num_proc=20` but having 19 files to download
- As by default, `DownloadManager` sets `num_proc=16`, before multiprocessing was only used when `len(iterable)>16` by default

Now, if `num_proc` is greater than or equal to ``len(iterable)``, `num_proc` is set to ``len(iterable)`` and multiprocessing is used.
- We pass the variable `multiprocessing_min_length=16`, so that multiprocessing is only used if at least 16 files to be downloaded
- ~As by default, `DownloadManager` sets `num_proc=16`, now multiprocessing is used when `len(iterable)>1` by default~

See discussion below.

~After having had to fix some tests (87602ac), I am wondering:~
- ~do we want to have multiprocessing by default?~
  - ~please note that `DownloadManager.download` sets `num_proc=16` by default~
- ~or would it be better to ask the user to set it explicitly if they want multiprocessing (and default to `num_proc=1`)?~

Fix #4636.

CC: @nateraw


Deprecate metrics

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4739). All of your documentation changes will be reflected on that endpoint.

I mark this as Draft because the deprecated version number needs being updated after the latest release.

Perhaps now is the time to also update the `inspect_metric` from `evaluate` with the changes introduced in https://github.com/huggingface/datasets/pull/4433 (cc @lvwerra)

What do you think of including what changes users have to do to switch to `evaluate` in the warning message ?
(basically replace `datasets.load_metric` by `evaluate.load`)

I think it can help users migrate to `evaluate` and silence the warnings

Deprecate metrics:
- deprecate public functions: `load_metric`, `list_metrics` and `inspect_metric`: docstring and warning
- test deprecation warnings are issues
- deprecate metrics in all docs
- remove mentions to metrics in docs and README
- deprecate internal functions/classes

Maybe we should also stop testing metrics?


Context: Ctenophores form an animal phylum that is more complex than sponges, about as complex as cnidarians (jellyfish, sea anemones, etc.), and less complex than bilaterians (which include almost all other animals). Unlike sponges, both ctenophores and cnidarians have: cells bound by inter-cell connections and carpet-like basement membranes; muscles; nervous systems; and some have sensory organs. Ctenophores are distinguished from all other animals by having colloblasts, which are sticky and adhere to prey, although a few ctenophore species lack them.

Question: Ctenophores are less complex than what other group?


Context: The motion picture, television, and music industry is centered on the Los Angeles in southern California. Hollywood, a district within Los Angeles, is also a name associated with the motion picture industry. Headquartered in southern California are The Walt Disney Company (which also owns ABC), Sony Pictures, Universal, MGM, Paramount Pictures, 20th Century Fox, and Warner Brothers. Universal, Warner Brothers, and Sony also run major record companies as well.

Question: What is the name given to the district that is associated with the motion picture industry?


Context: A second period of international expansion is linked to that of the ESPN network in the 1990s, and policies enacted in the 2000s by Disney Media Networks (which included the expansion of several of the company's U.S.-based cable networks including Disney Channel and its spinoffs Toon Disney, Playhouse Disney and Jetix; although Disney also sold its 33% stake in European sports channel Eurosport for $155 million in June 2000). In contrast to Disney's other channels, ABC is broadcast in the United States, although the network's programming is syndicated in many countries. The policy regarding wholly owned international networks was revived in 2004 when on September 27 of that year, ABC announced the launch of ABC1, a free-to-air channel in the United Kingdom owned by the ABC Group. However, on September 8, 2007, Disney announced that it would discontinue ABC1 citing to the channel's inability to attain sustainable viewership. With ABC1's shutdown that October, the company's attempt to develop ABC International were discontinued.

Question: When was ABC1 discontinued because of low viewership?

September 8, 2007
September 8, 2007
September 8, 2007


Context: Like mitochondria, chloroplasts use the potential energy stored in an H+, or hydrogen ion gradient to generate ATP energy. The two photosystems capture light energy to energize electrons taken from water, and release them down an electron transport chain. The molecules between the photosystems harness the electrons' energy to pump hydrogen ions into the thylakoid space, creating a concentration gradient, with more hydrogen ions (up to a thousand times as many) inside the thylakoid system than in the stroma. The hydrogen ions in the thylakoid space then diffuse back down their concentration gradient, flowing back out into the stroma through ATP synthase. ATP synthase uses the energy from the flowing hydrogen ions to phosphorylate adenosine diphosphate into adenosine triphosphate, or ATP. Because chloroplast ATP synthase projects out into the stroma, the ATP is synthesized there, in position to be used in the dark reactions.

Question: What does ATP synthase change into ATP?

phosphorylate adenosine diphosphate
phosphorylate adenosine diphosphate
energy from the flowing hydrogen ions


Context: There were many religions practiced during the Yuan dynasty, such as Buddhism, Islam, and Christianity. The establishment of the Yuan dynasty had dramatically increased the number of Muslims in China. However, unlike the western khanates, the Yuan dynasty never converted to Islam. Instead, Kublai Khan, the founder of the Yuan dynasty, favored Buddhism, especially the Tibetan variants. As a result, Tibetan Buddhism was established as the de facto state religion. The top-level department and government agency known as the Bureau of Buddhist and Tibetan Affairs (Xuanzheng Yuan) was set up in Khanbaliq (modern Beijing) to supervise Buddhist monks throughout the empire. Since Kublai Khan only esteemed the Sakya sect of Tibetan Buddhism, other religions became less important. He and his successors kept a Sakya Imperial Preceptor (Dishi) at court. Before the end of the Yuan dynasty, 14 leaders of the Sakya sect had held the post of Imperial Preceptor, thereby enjoying special power. Furthermore, Mongol patronage of Buddhism resulted in a number of monuments of Buddhist art. Mongolian Buddhist translations, almost all from Tibetan originals, began on a large scale after 1300. Many Mongols of the upper class such as the Jalayir and the Oronar nobles as well as the emperors also patronized Confucian scholars and institutions. A considerable number of Confucian and Chinese historical works were translated into the Mongolian language.

Question: What was Kublai's favorite sect of Tibetan Buddhism?


Context: In 1700 several hundred French Huguenots migrated from England to the colony of Virginia, where the English Crown had promised them land grants in Lower Norfolk County. When they arrived, colonial authorities offered them instead land 20 miles above the falls of the James River, at the abandoned Monacan village known as Manakin Town, now in Powhatan County. Some settlers landed in present-day Chesterfield County. On 12 May 1705, the Virginia General Assembly passed an act to naturalise the 148 Huguenots still resident at Manakintown. Of the original 390 settlers in the isolated settlement, many had died; others lived outside town on farms in the English style; and others moved to different areas. Gradually they intermarried with their English neighbors. Through the 18th and 19th centuries, descendants of the French migrated west into the Piedmont, and across the Appalachian Mountains into the West of what became Kentucky, Tennessee, Missouri, and other states. In the Manakintown area, the Huguenot Memorial Bridge across the James River and Huguenot Road were named in their honor, as were many local features, including several schools, including Huguenot High School.

Question: What town was actually granted to the Huguenots on arrival?


Context: Huguenot immigrants did not disperse or settle in different parts of the country, but rather, formed three societies or congregations; one in the city of New York, another 21 miles north of New York in a town which they named New Rochelle, and a third further upstate in New Paltz. The "Huguenot Street Historic District" in New Paltz has been designated a National Historic Landmark site and contains the oldest street in the United States of America. A small group of Huguenots also settled on the south shore of Staten Island along the New York Harbor, for which the current neighborhood of Huguenot was named.

Question: What is located within this district?


you know during the season and i guess at at your level uh you lose them to the next level if if they decide to recall the the parent team the Braves decide to call to recall a guy from triple A then a double A guy goes up to replace him and a single A guy goes up to replace him

You lose the things to the following level if the people recall.


One of our number will carry out your instructions minutely.

A member of my team will execute your orders with immense precision.

We have no one free at the moment so you have to take action yourself.


How do you know? All this is their information again.


yeah i tell you what though if you go price some of those tennis shoes i can see why now you know they're getting up in the hundred dollar range

The tennis shoes can be in the hundred dollar range.

The tennis shoes are not over hundred dollars.


my walkman broke so i'm upset now i just have to turn the stereo up real loud

I'm upset that my walkman broke and now I have to turn the stereo up really loud.

My walkman still works as well as it always did.


But a few Christian mosaics survive above the apse is the Virgin with the infant Jesus, with the Archangel Gabriel to the right (his companion Michael, to the left, has vanished save for a few feathers from his wings).

There are a few Christian mosaics left in the building.

There are no mosaics left in this place at all.


(Read  for Slate 's take on Jackson's findings.)

Slate had an opinion on Jackson's findings.

Slate did not hold any opinion on Jackson's findings.


At the end of Rue des Francs-Bourgeois is what many consider to be the city's most handsome residential square, the Place des Vosges, with its stone and red brick facades.

Place des Vosges is a very handsome area.

Place des Vosges is constructed entirely of gray marble.


I burst through a set of cabin doors, and fell to the ground-

I burst through the doors and fell down.


A group of kids is playing in a yard and an old man is standing in the background

A group of boys in a yard is playing and a man is standing in the background

A group of children playing in a yard, a man in the background.


A group of children is playing in the house and there is no man standing in the background

A group of kids is playing in a yard and an old man is standing in the background


The young boys are playing outdoors and the man is smiling nearby

The kids are playing outdoors near a man with a smile

The children are playing outdoors, while a man smiles nearby.


The kids are playing outdoors near a man with a smile


A brown dog is attacking another animal in front of the man in pants

A brown dog is jumping up at another dog in front of the man in jeans.


Nobody is riding the bicycle on one wheel

A person in a black jacket is doing tricks on a motorbike

A person rides their bicycle onto a rock and balances on one wheel.

A person in a black jacket doing tricks on a motorbike.


A person is riding the bicycle on one wheel

A man in a black jacket is doing tricks on a motorbike

A person in a black jacket doing tricks on a motorbike.

A person rides their bicycle onto a rock and balances on one wheel.


Working with one of the best Shakespeare sources, this film manages to be creditable to it's source, whilst still appealing to a wider audience.

Branagh steals the film from under Fishburne's nose, and there's a talented cast on good form.


Well...tremors I, the original started off in 1990 and i found the movie quite enjoyable to watch. however, they proceeded to make tremors II and III. Trust me, those movies started going downhill right after they finished the first one, i mean, ass blasters??? Now, only God himself is capable of answering the question "why in Gods name would they create another one of these dumpster dives of a movie?" Tremors IV cannot be considered a bad movie, in fact it cannot be even considered an epitome of a bad movie, for it lives up to more than that. As i attempted to sit though it, i noticed that my eyes started to bleed, and i hoped profusely that the little girl from the ring would crawl through the TV and kill me. did they really think that dressing the people who had stared in the other movies up as though they we're from the wild west would make the movie (with the exact same occurrences) any better? honestly, i would never suggest buying this movie, i mean, there are cheaper ways to find things that burn well.


Ouch! This one was a bit painful to sit through. It has a cute and amusing premise, but it all goes to hell from there. Matthew Modine is almost always pedestrian and annoying, and he does not disappoint in this one. Deborah Kara Unger and John Neville turned in surprisingly decent performances. Alan Bates and Jennifer Tilly, among others, played it way over the top. I know that's the way the parts were written, and it's hard to blame actors, when the script and director have them do such schlock. If you're going to have outrageous characters, that's OK, but you gotta have good material to make it work. It didn't here. Run away screaming from this movie if at all possible.


I've seen some crappy movies in my life, but this one must be among the very worst. Definately bottom 100 material (imo, that is).

We follow two couples, the Dodds (Billy Bob Thornton as Lonnie Earl and Natasha Richardson as Darlene) and the Kirkendalls (Patrick Swayze as Roy and Charlize Theron as Candy) in one car on a roadtrip to Reno.

Apparently, Lonnie isn't too happy with his sex-life, so he cheats on his wife with Candy, who's despirately trying to have a baby. Roy, meanwhile, isn't too sure if his sperm is OK so he's getting it checked by a doctor.

Now, I had read the back of the DVD, but my girlfriend didn't, and she blurted out after about 20 minutes: 'oh yeah, she's gonna end up pregnant but her husband can't have any baby's'. Spot on, as this movie is soooo predictable. As well as boring. And annoying. Meaningless. Offensive. Terrible.

An example of how much this movie stinks. The two couples set out in their big car towards Nevada, when they are stopped by 2 police-officers, as they didn't stop at a stop-sign. The guys know each other and finally bribe the two officers with a case of beer. Not only is this scene pointless and not important (or even relevant) for the movie, it takes about 5 minutes! It's just talk and talk and talk, without ever going somewhere.

I still have to puke thinking about the ending though. Apparently, Roy ISN'T having problems down there so he IS the father of the child. How many times does that happen in the movies... try something new! The cheated wife ultimately forgives her husband and best friend for having the affair and they all live happily ever after. Yuck.

Best scene of the movie is right at the end, with a couple of shots of the Grand Canyon. Why couldn't they just keep the camera on that for 90 minutes?

One would expect more from this cast (although Thornton really tries), but you can't really blame them. Writers, shame on you!

1/10.


I had been looking forward to seeing this film for a long time, after seeing "Return to Paradise," which I found to be gritty. I was so disappointed. The most realistic thing about it was the unpredictable ending which I think was partly stolen from "Return to Paradise."

Maybe I was expecting too much.

On the positive side Danes, Beckinsale and Pullman were fantastic in their roles. Although I didnt like Danes's character and first and found her very annoying.

I couldnt see anything realistic about the film. It could of been done so much better, for example there could of been more emphasis on the prison conditions and the sheer horror. It was too cheery a movie to be realistic. There could also of been more action and tension

The best thing about this film is the "tragic" ending. I couldnt of predicted that. But by that time I really didnt care what happened to them.

3/10


Effect(s) without cause is generally not possible in the real world but in the world of Hollywood remakes, not only is it possible, it's required. The Haunting has been given the computer treatment courtesy of a 1st-class cinematographer-turner-director who once showed promise (Jan de Bont- Speed) but has since produced a string of big budget garbage (Twister, Speed 2).

Actor are superfluous in a movie of this type and they seem to realize it. Liam Neeson and Cathrine Zeta-Jones act like they wish they were anywhere but in this film. Lili Taylor makes an attempt to add something to the proceedings but whatever that something might be is unknown since the script feels like half of it is missing. Events just happen, good and bad ghosts show up with no rhyme or reason and then the story just ends with a most unsatisfying non-event meant to wrap up the previous 90 minutes of inanity.

There really isn't even reason to see this for the effects since we all know that anything can be put on screen now. Why not watch effects in the service of a good story instead of just for their own sake?


This picture started out with good intentions, Bacon the scientist out to test the theory of invisibility, and Shue is cute as usual in her role. It all falls apart after that, it's your typical Hollywood thriller now, filmed on a soundstage with special effects galore, minus any kind of humour, wit or soul. In other words, don't waste your time watching this. Get the audiocassette tape with John DeLancie as the Invisible Man instead, also starring Leonard Nimoy. Now that was good, and HG Wells is well served, unlike with this mess.


I chose to see this movie because it got a good score here on IMDb. But a lot of people either have really poor taste or someone's been fixing the score.

Either way it was a real disappointment. The movie is exactly as stupid and far fetched as the title would suggest. There really is no reason to give a summary of the plot - but here goes: it felt like someone had been thinking: "Wouldn't it be cool to make a movie where there were snakes on a plane? And then the snakes for some reason would go crazy and start biting and stuff?!?" And that's about it! The plot is thin and unoriginal. The snakes are bad CGI (but it makes sense to cut corners on a movie that no one in their right mind will recommend to anyone!). The acting is poor, and all people are unbelievable stereo types.

To sum it up: It's one of the worst movies I've ever seen - stay away!


This film has to be the worst I have ever seen. The title of the film deceives the audience into thinking there maybe hope. The story line of the film is laughable at best, with the acting so poor you just have to cringe. The title 'Zombie Nation' implies a hoard of zombies when in fact there are six in total. This cannot be categorised as a horror film due to the introduction of cheesy 80's music when the zombies 'attack'. The zombies actually talk and act like human beings in the film with the only difference being the make up which looks like something out a La Roux video. If you ever get the chance to buy this film then do so, then burn the copy.


Donna: Bohemian Rapsody Central Cinema tomorrow 8pm
Donna: Are u in?
Michael: totally!
Michael: great idea
Donna: OK, see you in front of the main entrance at 7:50

Tomorrow Donna and Michael will meet in front of the main entrance of Central Cinema at 7:50 pm to watch Bohemian Rapsody at 8.


John: Anything from the store?
Tracy: Hmmm... there was this goat cheese last time that I really liked. You remember which one.
John: I think I do. Anything else?
Tracy: Maybe milk, cause we're running out. And like a couple of rolls, the grainy ones.
Tracy: oh and tissues.
John: Ok, roger that.

John will buy the goat cheese Tracy liked, milk, a couple of grainy rolls, and tissues.


Jimmy: so how is the dispute between your countries going? hihi
Jimmy: I think it's a never-ending epic
Jose: Actually, it seems it's better now
Jimmy: is it?
Dean: yup. they held a referendum in Guatemala last year
Jimmy: only Guatemala? I thought the referendums were supposed to be held simultaneously
Hugo: yes, but they decided to change it
Jimmy: so what about Belize?
Hugo: Belize will hold a referendum in April 2019
Dean: at least it's the official plan
Jimmy: wow, that's amazing after a few centuries hahaha
Dean: true, but it's not settling the dispute, they will just send the case to the international tribunal
Jimmy: I see, it's interesting

Two countries haven't been able to reach agreement for centuries. There was a referendum in Guatemala last year. According to the plan, Belize will hold a referendum in April 2019. However, it's not settling the dispute and they'll send the case to the international tribunal.


Ben: I'm waiting downstairs
Tom: Adam is shitting
Adam: Give me 5 more minutes!!!


Jimmy: get me some fries too!!
Dan: okay, ketchup?
Jimmy: and mayo
Dan: ok!

Dan will get Jimmy some fries with ketchup and mayo.


Eve: Has anyone seen the new HP film?
Alice: I saw it with Phillip yesterday
Eve: And, how did you like it?
Alice: I really liked it. Phillip a lot less :P
Carla: No spoilers! I am planning on going this weekend
Eve: Great, I am going with a friend from my studies. Care to join us?
Carla: Saturday or Sunday?
Eve: We aren't sure yet, but we want 2D, so not a lot of choice. Maybe Saturday at 16:50?
Carla: Works for me :) Later we can go to the main square for some hot wine?
Eve: Great idea ;) Will you join us Alice?
Alice: I wish I could, but I am going to that wedding I mentioned last time :(
Carla: Uuuu, be sure to find me a future husband among Phillips family ;)
Alice: As far as I know we will be the youngest couple there :p
Carla: I don't mind some old and wealthy guy ;)

Phillip and Alice saw the new HP film yesterday. She liked it more than him. Eve is going to see it with friends from studies on Saturday at 16:50. Carla will join them. Alice can't go because she is going to the wedding.


Eva: I've signed the agreement
Tina: You did? omg so happy for ya
Eva: Thanks:) I am super excited
Barb: Omg babe! awesome!
Eva: hahaha thanks! I didn't know it's gonna be so emotional!
Barb: Hell yeah! It's kinda binding for 25 years you know, kinda scary
Eva: I know, but, Mat and I are getting married so after that we're gonna consolidate or sth and pay it off together faster
Barb: You guys are getting married?
Eva: Yes we are! you didn't know?
Barb: First time hearing this
Tina: No worries I just found out myself
Barb: Anyway! Congrats love! You guys are awesome!
Eva: Thank you so much! It's an exciting time
Barb: Well I hope we're all invited for the housewarming party
Eva: Of course! I'll be sending emails

Eva signed the agreement. She and Mat are getting married. They will organize the housewarming party in their new flat.


John: I'm starving. What's for dinner?
Jane: Whatever you cook :-)
John: Damn! I best get some take-aways.


Elena: Hi! What's up?
Ann: Nothing much, I'm working.
Elena: Oh, I see. When will you be free?
Ann: I'm off at 5. Wanna grab some coffee?
Elena: Yeah, it's been ages, I have to talk to you.
Ann: Great, is everything OK?
Elena: Yeah, it's just Robert, he's acting weird.
Ann: Again? I thought you guys figured everything out.
Elena: I thought so too... But he keeps getting these "thoughts", you know. About the future and stuff and he acts weirder than normal.
Ann: Yeah, he's always been like that, right? Getting worked up over nothing.
Elena: That's the thing. He gets mad over some things he made up in his mind and he blames me for it!
Ann: You have to be patient, you know what he went through last year.
Elena: I know, but I stood by him, every step of the way. And he acts as if I didn't understand.
Ann: Honey, I'm sure it will all get better, we'll talk it through this afternoon, but I really need to get back to work, sorry.
Elena: Sure thing, sorry I kept you.
Ann: Don't mention it, it's just my boss is looking.
Elena: LOL, tell him you're counseling your crazy friend.
Ann: I'm not sure it's the best way to suck up to your boss xD
Elena: haha, I figure it's not. See you!

Ann and Elena arrange meeting for a coffee around 5, after Anna has finished work. Elena would like to talk about Robert, as he's acting weird, having 'thoughts' about the future and blaming her for some things he made up. She needs to be very patient with him.


get

Open screens/snackbars/dialogs without context, manage states and inject dependencies easily with GetX.


provider

A wrapper around InheritedWidget to make them easier to use and more reusable.


shared_preferences

flutter, shared_preferences_android, shared_preferences_ios, shared_preferences_linux, shared_preferences_macos, shared_preferences_platform_interface, shared_preferences_web, shared_preferences_windows

Flutter plugin for reading and writing simple key-value pairs. Wraps NSUserDefaults on iOS and SharedPreferences on Android.


http

A composable, multi-platform, Future-based API for HTTP requests.


url_launcher

flutter, url_launcher_android, url_launcher_ios, url_launcher_linux, url_launcher_macos, url_launcher_platform_interface, url_launcher_web, url_launcher_windows

Flutter plugin for launching a URL. Supports web, phone, SMS, and email schemes.


flutter_bloc

Flutter Widgets that make it easy to implement the BLoC (Business Logic Component) design pattern. Built to be used with the bloc state management package.


dio

A powerful Http client for Dart, which supports Interceptors, FormData, Request Cancellation, File Downloading, Timeout etc.


flutter_launcher_icons

A package which simplifies the task of updating your Flutter app's launcher icon.


image_picker

flutter, image_picker_android, image_picker_for_web, image_picker_ios, image_picker_platform_interface

Flutter plugin for selecting images from the Android and iOS image library, and taking new pictures with the camera.


animations

Fancy pre-built animations that can easily be integrated into any Flutter application.


http :// www . bangkokhealth . com / healthnews _ htdoc / healthnews _ detail . asp ? Number = 10506


TITLE
Add LLaVA model

COMMENTS
11

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### Model description

[LLaVA](https://llava-vl.github.io/) is a multimodal model that combines a vision encoder and Vicuna for general-purpose visual and language understanding, "achieving impressive chat capabilities mimicking spirits of the multimodal GPT-4".

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/haotian-liu/LLaVA


TITLE
XGLMForCausalLM does not support `device_map='auto'` for load 8 bit

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

transformers: v4.27.0

### Who can help?

@sgugger @muellerzr

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I was use this code.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "facebook/xglm-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, load_in_8bit=True, device_map='auto')
```

Error:

```python
Overriding torch_dtype=None with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 3
      1 model_name = "facebook/xglm-1.7B"
      2 tokenizer = AutoTokenizer.from_pretrained(model_name)
----> 3 model_8bit = AutoModelForCausalLM.from_pretrained(model_name, load_in_8bit=True, device_map='auto')

File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:471, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    469 elif type(config) in cls._model_mapping.keys():
    470     model_class = _get_model_class(config, cls._model_mapping)
--> 471     return model_class.from_pretrained(
    472         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    473     )
    474 raise ValueError(
    475     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    476     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    477 )

File /usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py:2556, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2550 special_dtypes = {
   2551     name: torch.float32
   2552     for name, _ in model.named_parameters()
   2553     if any(m in name for m in keep_in_fp32_modules)
   2554 }
   2555 if model._no_split_modules is None:
-> 2556     raise ValueError(f"{model.__class__.__name__} does not support `device_map='{device_map}'` yet.")
   2557 no_split_modules = model._no_split_modules
   2558 if device_map not in ["auto", "balanced", "balanced_low_0", "sequential"]:

ValueError: XGLMForCausalLM does not support `device_map='auto'` yet.
```

### Expected behavior

XGLMForCausalLM should support `device_map='auto'`.


TITLE
bug: eval_accumulation_steps can lead to incorrect metrics

COMMENTS
3

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.11.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

Hey @sgugger, I'm tagging you since this has to do with the trainer.

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using the `run_qa.py` script in the `examples/pytorch/question-answering/` folder
```bash
python run_qa.py \
  --model_name_or_path "sjrhuschlee/flan-t5-base-squad2" \
  --dataset_name squad_v2 \
  --output_dir "tmp/eval_squad_v2/" \
  --version_2_with_negative True \
  --max_seq_length 512 \
  --doc_stride 128 \
  --do_eval \
  --per_device_eval_batch_size 24 \
  --tf32 True \
  --dataloader_num_workers 6 \
  --preprocessing_num_workers 6 \
  --bf16_full_eval \
  --eval_accumulation_steps 2 \
  --overwrite_output_dir False
```
I found that the calculated metrics when using `eval_accumulation_steps` is not always correct. When not using `eval_accumulation_steps` with the above script I find that I get the expected metrics. However, I found that I needed to use `eval_accumulation_steps` for evaluation of the `flan-t5` models with the above parameters on my system otherwise the memory usage on the GPU would fluctuate from 4 - 8GB which could cause an OOM.

I believe I found the cause for the inconsistency in the metrics. Specifically this line https://github.com/huggingface/transformers/blob/a074a5d34d6411fb00e83a2ed30acf23d8c976b5/src/transformers/trainer.py#L3150
does not cover the edge case where the total number of batches in the evaluation is not exactly divisible by `eval_accumulation_steps`. For example, if `eval_accumulation_steps = 2` and the total number of batches is 613, then only the last batch is used when calculating `all_preds`.

I was able to partially fix this problem by adding a new variable called `total_steps` and updating the if statement
```python
logger.info(f"***** Running {description} *****")
if has_length(dataloader):
    total_steps = len(dataloader)
    logger.info(f"  Num examples = {self.num_examples(dataloader)}")
else:
    total_steps = None
    logger.info("  Num examples: Unknown")

...

  if args.eval_accumulation_steps is not None and (
          (step + 1) % args.eval_accumulation_steps == 0 or (step + 1) == total_steps
  ):
```
However, this will still be a problem for dataloaders that don't have a defined length.

### Expected behavior

Using `eval_accumulation_steps` should work in every case even when the number of batches is not divisible by `eval_accumulation_steps`.


TITLE
Add EnCodec model

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# What does this PR do?

Adds the EnCodec neural codec from the [High Fidelity Neural Audio Compression](https://arxiv.org/abs/2210.13438) paper.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


TITLE
Support gradient checkpointing for ESM models

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Would you please add `gradient_checkpointing_enable()` feature for ESM models?
These models currently are the best available pre-trained protein language models for researchers.
Many thanks.


TITLE
Add MobileViT v2

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### Model description

[MobileViT](https://openreview.net/forum?id=vh-0sUt8HlG) is a computer vision model that combines CNNs with transformers that has already been added to Transformers.

[MobileViT v2](https://arxiv.org/abs/2206.02680) is the second version; it is constructed by replacing multi-headed self-attention in MobileViT v1 with the proposed separable self-attention.

Does Hugging Face have plan to add MobileViT v2 to Transformers?

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The official implementation is from Apple at this link: [https://github.com/apple/ml-cvnets](https://github.com/apple/ml-cvnets)

The timm library also implemented it and has pre-trained weights at this link: [https://github.com/huggingface/pytorch-image-models/blob/82cb47bcf360e1974c00c35c2aa9e242e6b5b565/timm/models/mobilevit.py](https://github.com/huggingface/pytorch-image-models/blob/82cb47bcf360e1974c00c35c2aa9e242e6b5b565/timm/models/mobilevit.py)