Catch pull error when mirroring

Hi @lewtun, thanks for reporting.

It seems that our library fails at inferring the dtype of the columns:
- `milestone`
- `performed_via_github_app`

(and assigns them `null` dtype).

Catch pull errors when mirroring so that the script continues to update the other datasets.

The error will still be printed at the end of the job. In this case the job also fails, and asks to manually update the datasets that failed.


Add API code examples for Builder classes

documentation

Improvements or additions to documentation

This PR adds API code examples for the Builder classes.


added TR-News dataset


[Imagefolder] Docs + Don't infer labels from file names when there are metadata + Error messages when metadata and images aren't linked correctly

I updated the `docs/source/image_process.mdx` documentation and added an example for image captioning and object detection using `ImageFolder`.

While doing so I also improved a few aspects:
- we don't need to infer labels from file names when there are metadata - they can just be in the metadata if necessary
- raise informative error messages when metadata and images aren't linked correctly:
  - when an image is missing a metadata file
  - when a metadata file is missing an image

I added some tests for these changes as well

cc @mariosasko


Loading dataset with streaming: '_io.BufferedReader' object has no attribute 'loc'

bug

Something isn't working

## Describe the bug
Loading a datasets with `load_dataset` and `streaming=True` returns `AttributeError: '_io.BufferedReader' object has no attribute 'loc'`. Notice that loading with `streaming=False` works fine.

In the following steps we load parquet files but the same happens with pickle files. The problem seems to come from `fsspec` lib, I put in the environment info also `s3fs` and `fsspec` versions since I'm loading from an s3 bucket.

## Steps to reproduce the bug
```python
from datasets import load_dataset
# path is the path to parquet files
data_files = {"train": path + "meta_train.parquet.gzip", "test": path + "meta_test.parquet.gzip"}
dataset = load_dataset("parquet", data_files=data_files, streaming=True)
```

## Expected results
A dataset object `datasets.dataset_dict.DatasetDict`

## Actual results
```
AttributeError                            Traceback (most recent call last)
<command-562086> in <module>
     11
     12 data_files = {"train": path + "meta_train.parquet.gzip", "test": path + "meta_test.parquet.gzip"}
---> 13 dataset = load_dataset("parquet", data_files=data_files, streaming=True)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1679     if streaming:
   1680         extend_dataset_builder_for_streaming(builder_instance, use_auth_token=use_auth_token)
-> 1681         return builder_instance.as_streaming_dataset(
   1682             split=split,
   1683             use_auth_token=use_auth_token,

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/builder.py in as_streaming_dataset(self, split, base_path, use_auth_token)
    904         )
    905         self._check_manual_download(dl_manager)
--> 906         splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
    907         # By default, return all splits
    908         if split is None:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/packaged_modules/parquet/parquet.py in _split_generators(self, dl_manager)
     30         if not self.config.data_files:
     31             raise ValueError(f"At least one data file must be specified, but got data_files={self.config.data_files}")
---> 32         data_files = dl_manager.download_and_extract(self.config.data_files)
     33         if isinstance(data_files, (str, list, tuple)):
     34             files = data_files

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in download_and_extract(self, url_or_urls)
    798
    799     def download_and_extract(self, url_or_urls):
--> 800         return self.extract(self.download(url_or_urls))
    801
    802     def iter_archive(self, urlpath_or_buf: Union[str, io.BufferedReader]) -> Iterable[Tuple]:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in extract(self, path_or_paths)
    776
    777     def extract(self, path_or_paths):
--> 778         urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
    779         return urlpaths
    780

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm, desc)
    312         num_proc = 1
    313     if num_proc <= 1 or len(iterable) <= num_proc:
--> 314         mapped = [
    315             _single_map_nested((function, obj, types, None, True, None))
    316             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    313     if num_proc <= 1 or len(iterable) <= num_proc:
    314         mapped = [
--> 315             _single_map_nested((function, obj, types, None, True, None))
    316             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    317         ]

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    267         return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
    268     else:
--> 269         mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
    270         if isinstance(data_struct, list):
    271             return mapped

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    267         return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
    268     else:
--> 269         mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
    270         if isinstance(data_struct, list):
    271             return mapped

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    249     # Singleton first to spare some computation
    250     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 251         return function(data_struct)
    252
    253     # Reduce logging to keep things readable in multiprocessing with tqdm

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _extract(self, urlpath)
    781     def _extract(self, urlpath: str) -> str:
    782         urlpath = str(urlpath)
--> 783         protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
    784         if protocol is None:
    785             # no extraction

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _get_extraction_protocol(urlpath, use_auth_token)
    371         urlpath, kwargs = urlpath, {}
    372     with fsspec.open(urlpath, **kwargs) as f:
--> 373         return _get_extraction_protocol_with_magic_number(f)
    374
    375

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _get_extraction_protocol_with_magic_number(f)
    335 def _get_extraction_protocol_with_magic_number(f) -> Optional[str]:
    336     """read the magic number from a file-like object and return the compression protocol"""
--> 337     prev_loc = f.loc
    338     magic_number = f.read(MAGIC_NUMBER_MAX_LENGTH)
    339     f.seek(prev_loc)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/fsspec/implementations/local.py in __getattr__(self, item)
    337
    338     def __getattr__(self, item):
--> 339         return getattr(self.f, item)
    340
    341     def __enter__(self):

AttributeError: '_io.BufferedReader' object has no attribute 'loc'
```
## Environment info
- `datasets` version: 2.1.0
- Platform: Linux-5.4.0-1071-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
- `fsspec` version: 2021.08.1
- `s3fs` version: 2021.08.1


[WIP] Add TEDLIUM dataset

dataset request

Requesting to add a new dataset

Adds the TED-LIUM dataset https://www.tensorflow.org/datasets/catalog/tedlium#tedliumrelease3

TODO:

- [x] Port `tedium.py` from TF datasets using `convert_dataset.sh` script
- [ ] Make `load_dataset` work
- [ ] Run `datasets-cli` command to generate `dataset_infos.json`
- [ ] Create dummy data for continuous testing
- [ ] Dummy data tests
- [ ] Real data tests
- [ ] Create the metadata JSON
- [ ] Close PR and add directly to the Hub under LIUM org


Remove unused multiprocessing args from test CLI

Multiprocessing is not used in the test CLI.


Add packaged builder configs to the documentation

Add the packaged builders configurations to the docs reference is useful to show the list of all parameters one can use when loading data in many formats: CSV, JSON, etc.


Fixes FrugalScore

There are two minor modifications in this PR:
1) `predictions` and `references` are swapped. Basically Frugalscore is commutative, however some tiny differences can occur if we swap the references and the predictions. I decided to swap them just to obtain the exact results as reported in the paper.
2) I switched to dynamic padding that was was used in the training, forcing the padding to `max_length`  introduces errors for some reason that I ignore.

@lhoestq


plot : two teen couples go to a church party , drink and then drive .
they get into an accident .
one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares .
what 's the deal ?
watch the movie and " sorta " find out . . .
critique : a mind - fuck movie for the teen generation that touches on a very cool idea , but presents it in a very bad package .
which is what makes this review an even harder one to write , since i generally applaud films which attempt to break the mold , mess with your head and such ( lost highway & memento ) , but there are good and bad ways of making all types of films , and these folks just did n't snag this one correctly
.
they seem to have taken this pretty neat concept , but executed it terribly .
so what are the problems with the movie ?
well , its main problem is that it 's simply too jumbled
.
it starts off " normal " but then downshifts into this " fantasy " world in which you , as an audience member ,
have no idea what 's going on
.
there are dreams , there are characters coming back from the dead , there are others who look like the dead , there are strange apparitions , there are disappearances , there are a looooot of chase scenes , there are tons of weird things that happen , and most of it is simply not explained .
now i personally do n't mind trying to unravel a film every now and then , but when all it does is give me the same clue over and over again , i get kind of fed up after a while , which is this film 's biggest problem .
it 's obviously got this big secret to hide , but it seems to want to hide it completely until its final five minutes .
and do they make things entertaining , thrilling or even engaging , in the meantime ?
not really .
the sad part is that the arrow and i both dig on flicks like this , so we actually figured most of it out by the half - way point , so all of the strangeness after that did start to make a little bit of sense , but it still did n't the make the film all that more entertaining .
i guess the bottom line with movies like this is that you should always make sure that the audience is " into it " even before they are given the secret password to enter your world of understanding .
i mean , showing melissa sagemiller running away from visions for about 20 minutes throughout the movie is just plain lazy ! !
okay , we get it .
. .
there are people chasing her and we do n't know who they are .
do we really need to see it over and over again ?
how about giving us different scenes offering further insight into all of the strangeness going down in the movie ?
apparently , the studio took this film away from its director and chopped it up themselves , and it shows .
there might 've been a pretty decent teen mind - fuck movie in here somewhere , but i guess " the suits " decided that turning it into a music video with little edge , would make more sense .
the actors are pretty good for the most part , although wes bentley just seemed to be playing the exact same character that he did in american beauty , only in a new neighborhood .
but my biggest kudos go out to sagemiller , who holds her own throughout the entire film , and actually has you feeling her character 's unraveling .
overall , the film does n't stick
because it does n't entertain , it 's confusing , it rarely excites and
it feels pretty redundant for most of its runtime , despite a pretty cool ending and explanation to all of the craziness that came before it .
oh ,
and by the way , this is not a horror or teen slasher flick . . .
it 's just packaged to look that way because someone is apparently assuming that the genre is still hot with the kids .
it also wrapped production two years ago and has been sitting on the shelves ever since .
whatever . .
. skip it !
where 's joblo coming from ?
a nightmare of elm street 3 ( 7/10 ) - blair witch 2 ( 7/10 ) - the crow ( 9/10 ) - the crow : salvation ( 4/10 )
- lost highway ( 10/10 ) - memento ( 10/10 ) - the others ( 9/10 ) - stir of echoes ( 8/10 )


the happy bastard 's quick movie review damn
that y2k bug .
it 's got a head start in this movie starring jamie lee curtis and another baldwin brother ( william this time ) in a story regarding a crew of a tugboat that comes across a deserted russian tech ship that has a strangeness to it when they kick the power back on .
little do they know the power within . . .
going for the gore and bringing on a few action sequences here and there , virus
still feels very empty , like a movie going for all flash and no substance .
we do n't know why the crew was really out in the middle of nowhere , we do n't know the origin of what took over the ship ( just that a big pink flashy thing hit the mir ) , and , of course , we do n't know why donald sutherland is stumbling around drunkenly throughout .
here , it 's just
" hey , let 's chase these people around with some robots " .
the acting is below average , even from the likes of curtis .
you 're more likely to get a kick out of her work in halloween h20 .
sutherland is wasted and baldwin , well , he 's acting like a baldwin , of course .
the real star here are stan winston 's robot design , some schnazzy cgi , and the occasional good gore shot , like picking into someone 's brain .
so , if robots and body parts really turn you on , here 's your movie .
otherwise , it 's pretty much a sunken ship of a movie .

it 's pretty much a sunken ship
sutherland is wasted
still feels very empty
the acting is below average


it is movies like these that make a jaded movie viewer thankful for the invention of the timex indiglo watch .
based on the late 1960 's television show by the same name , the mod squad tells the tale of three reformed criminals under the employ of the police to go undercover .
however , things go wrong as evidence gets stolen and they are immediately under suspicion .
of course , the ads make it seem like so much more .
quick cuts , cool music , claire dane 's nice hair and cute outfits , car chases , stuff blowing up , and the like . sounds like a cool movie ,
does it not ?
after the first fifteen minutes , it quickly becomes apparent that it is not .
the mod squad is certainly a slick looking production , complete with nice hair and costumes , but that simply is n't enough .
the film is best described as a cross between an hour - long cop show and a music video , both stretched out into the span of an hour and a half . and
with it comes every single clich ?
.
it does n't really matter that the film is based on a television show , as most of the plot elements have been recycled from everything we 've already seen .
the characters and acting is nothing spectacular , sometimes even bordering on wooden .
claire danes and omar epps deliver their lines as if they are bored , which really transfers onto the audience .
the only one to escape relatively unscathed is giovanni ribisi , who plays the resident crazy man , ultimately being the only thing worth watching .
unfortunately , even he 's not enough to save this convoluted mess , as
all the characters do n't do much apart from occupying screen time .
with the young cast , cool clothes , nice hair , and hip soundtrack , it appears that the film is geared towards the teenage mindset .
despite an american ' r ' rating ( which the content does not justify ) , the film is way too juvenile for the older mindset .
information on the characters is literally spoon - fed to the audience ( would it be that hard to show us instead of telling us ? ) , dialogue is poorly written , and the plot is extremely predictable .
the way the film progresses , you likely wo n't even care if the heroes are in any jeopardy , because you 'll know they are n't . basing the show on a 1960 's television show that nobody remembers is of questionable wisdom , especially when one considers the target audience and the fact that the number of memorable films based on television shows can be counted on one hand ( even one that 's missing a finger or two ) .
the number of times that i checked my watch ( six ) is a clear indication that this film is not one of them .
it is clear that the film is nothing more than an attempt to cash in on the teenage spending dollar , judging from the rash of really awful teen - flicks that we 've been seeing as of late . avoid this film at all costs .


" quest for camelot " is warner bros . '
first feature - length , fully - animated attempt to steal clout from disney 's cartoon empire , but the mouse has no reason to be worried .
the only other recent challenger to their throne was last fall 's promising , if flawed , 20th century fox production " anastasia , " but disney 's " hercules , " with its lively cast and colorful palate , had her beat hands - down when it came time to crown 1997 's best piece of animation .
this year , it 's no contest , as " quest for camelot " is pretty much dead on arrival .
even the magic kingdom at its most mediocre
-- that 'd be " pocahontas " for those of you keeping score -- is n't nearly as dull as this .
the story revolves around the adventures of free - spirited kayley ( voiced by jessalyn gilsig ) , the early - teen daughter of a belated knight from king arthur 's round table .
kayley 's only dream is to follow in her father 's footsteps , and she gets her chance when evil warlord ruber ( gary oldman ) , an ex - round table member - gone - bad , steals arthur 's magical sword excalibur and accidentally loses it in a dangerous , booby - trapped forest .
with the help of hunky , blind timberland - dweller garrett ( carey elwes ) and a two - headed dragon ( eric idle and don rickles ) that 's always arguing with itself , kayley just might be able to break the medieval sexist mold and prove her worth as a fighter on arthur 's side .
" quest for camelot " is missing pure showmanship , an essential element if it 's ever expected to climb to the high ranks of disney .
there 's nothing here
that differentiates " quest " from something you 'd see on any given saturday morning cartoon -- subpar animation , instantly forgettable songs , poorly - integrated computerized footage .
( compare kayley and garrett 's run -
in with the angry ogre
to herc 's battle with the hydra .
i rest my case . )
even the characters stink -- none of them are remotely interesting , so much that the film becomes a race to see which one can out - bland the others .
in the end , it 's a tie -- they all win .
that dragon 's comedy shtick is awfully cloying , but at least it shows signs of a pulse .
at least fans of the early-'90s tgif television line - up will be thrilled to find jaleel " urkel " white and bronson " balki " pinchot sharing the same footage .
a few scenes are nicely realized ( though i 'm at a loss to recall enough to be specific ) , and the actors providing the voice talent are enthusiastic ( though most are paired up with singers who do n't sound a thing like them for their big musical moments -- jane seymour and celine dion ? ? ? ) .
but one must strain through too much of this mess to find the good . aside from the fact that children will probably be as bored watching this as adults , " quest for camelot
" 's most grievous error is its complete lack of personality .
and personality , we learn from this mess , goes a very long way .


synopsis : a mentally unstable man undergoing psychotherapy saves a boy from a potentially fatal accident and then falls in love with the boy 's mother , a fledgling restauranteur .
unsuccessfully attempting to gain the woman 's favor , he takes pictures of her and kills a number of people in his way .
comments : stalked is yet another in a seemingly endless string of spurned - psychos - getting - their - revenge type movies which are a stable category in the 1990s film industry , both theatrical and direct - to - video .
their proliferation may be due in part to the fact that they 're typically inexpensive to produce ( no special effects , no big name stars ) and serve as vehicles to flash nudity ( allowing them to frequent late - night cable television ) .
stalked wavers slightly from the norm in one respect : the psycho never actually has an affair ; on the contrary , he 's rejected rather quickly ( the psycho typically is an ex - lover , ex - wife , or ex - husband ) .
other than that , stalked is just another redundant entry doomed to collect dust on video shelves and viewed after midnight on cable .
stalked does not provide much suspense , though that is what it sets out to do .
interspersed throughout the opening credits , for instance , a serious - sounding narrator spouts statistics about stalkers and ponders what may cause a man to stalk ( it 's implicitly implied that all stalkers are men ) while pictures of a boy are shown on the screen .
after these credits , a snapshot of actor jay underwood appears .
the narrator states that " this is the story of daryl gleason " and tells the audience that he is the stalker .
of course , really , this is the story of restauranteur brooke daniels .
if the movie was meant to be about daryl , then it should have been called stalker not stalked .
okay .
so we know who the stalker is even before the movie starts ; no guesswork required .
stalked proceeds , then , as it begins : obvious , obvious , obvious .
the opening sequence , contrived quite a bit , brings daryl and brooke ( the victim ) together .
daryl obsesses over brooke , follows her around , and tries to woo her .
ultimately rejected by her , his plans become more and more desperate and elaborate .
these plans include the all - time , psycho - in - love , cliche : the murdered pet .
for some reason , this genre 's films require a dead pet to be found by the victim stalked .
stalked is no exception ( it 's a cat this time -- found in the shower ) .
events like these lead to the inevitable showdown between stalker and stalked , where only one survives ( guess who it invariably always is and you 'll guess the conclusion to this turkey ) .
stalked 's cast is uniformly adequate : not anything to write home about but also not all that bad either .
jay underwood , as the stalker , turns toward melodrama a bit too much .
he overdoes it , in other words , but he still manages to be creepy enough to pass as the type of stalker the story demands .
maryam d'abo , about the only actor close to being a star here
( she played the bond chick in the living daylights ) , is equally adequate as the " stalked " of the title , even though she seems too ditzy at times to be a strong , independent business - owner .
brooke ( d'abo ) needs to be ditzy , however , for the plot to proceed .
toward the end , for example , brooke has her suspicions about daryl .
to ensure he wo n't use it as another excuse to see her , brooke decides to return a toolbox he had left at her place to his house .
does she just leave the toolbox at the door when no one answers ?
of course not .
she tries the door , opens it , and wanders around the house .
when daryl returns , he enters the house , of course , so our heroine is in danger .
somehow , even though her car is parked at the front of the house , right by the front door , daryl is oblivious to her presence inside .
the whole episode places an incredible strain on the audience 's suspension of disbelief and questions the validity of either character 's intelligence .
stalked receives two stars because , even though
it is highly derivative and somewhat boring
, it is not so bad that it can not be watched .
rated r mostly for several murder scenes and brief nudity in a strip bar , it is not as offensive as many other thrillers in this genre are .
if you 're in the mood for a good suspense film , though , stake out something else .


capsule : in 2176 on the planet mars police taking into custody an accused murderer face the title menace .
there is a lot of fighting and not a whole lot of story otherwise .
john carpenter reprises so many ideas from his previous films , especially assault on precinct 13 , that the new film comes off as his homage to himself .
, 0 ( -4 to +4 ) .
john carpenter apparently believes that action scenes in which people fight something horrible are the same as horror scenes .
for a writer and director of horror films , supposedly an expert on horror ,
it is a very bad mistake to make .
ghosts of mars is called a horror movie , but it is more just a drawn out fight between humans and a surprisingly low - powered alien menace .
in addition if anybody but john carpenter had made ghosts of mars , carpenter would have grounds to sue .
this film is just chock full of pieces taken from assault on precinct 13 , the thing , and prince of darkness .
it is , in fact , surprising that carpenter managed to fit so many pieces of his previous work into this film in such an admittedly novel way .
but that still does not make for a really good science fiction experience .
ghosts of mars takes place in the year 2176 .
mars has been mostly terraformed so that humans can walk on the surface without breathing gear ( which is good for the film 's budget ) .
it is never mentioned , but the gravity on mars has been increased somehow to earth - normal , again making it easier to film .
society has changed a bit by that time , but it has advanced surprisingly little .
apparently the culture has changed so that women are much more in positions of control .
and from carpenter 's view , women have really made a mess of things .
society has stagnated under female control so that beyond some minor technological advances society has changed less in 175 years than we might expect it to change in ten .
the basic plot of ghosts of mars has much in common with that of assault on precinct 13 except that precinct 9 ( yes , precinct 9 ) has been replaced by a somewhat tacky looking rundown martian mining colony .
instead of having the criminal " napolean " wilson , this film has the criminal " desolation " williams .
instead of facing hoodlums with automatic weapons the police face , well , ghosts of mars .
because the ghosts are somewhat alien in nature they should behave in some alien manner , but they essentially behave as human savages , in another lapse of imagination .
the story is told in flashback , flashback within flashback , and flashback within flashback within flashback .
ghosts of mars takes place entirely at night and is filmed almost entirely in tones of red , yellow , and black .
carpenter manages to give us a powerful opening scene , showing a mining train rushing through the martian night to the sound of music with a heavy beat .
sadly what follows is not really up to the buildup .
the terror he creates looks a little too much like fugitive wannabes from the rock band kiss .
his idea of building suspense is having a bunch of sudden jump scenes that sucker the viewer into thinking something scary is happening and then prove to be just something boring .
these are standard haunted house film shock effects that require no great talent to give the audience .
somewhat newer but also unimpressive
are the cgi digital decapitations in some of the fights .
within a short stretch of time we have seen the release of mission to mars , red planet , and ghosts of mars .
after mission to mars was panned by too many reviewers it looks better and better and better as time goes by .
i rate ghosts of mars a 4 on the 0 to 10 scale and a 0 on the -4 to +4 scale .
following the movie i showed my wife , who liked ghosts of mars moderately more than i did , carpenter 's classic assault on precinct 13 .
her comment is that it was seeing the same film twice .


so ask yourself what " 8 mm " ( " eight millimeter " ) is really all about .
is it about a wholesome surveillance man who loses sight of his values after becoming enmeshed in the seedy , sleazy underworld of hardcore pornography ?
is it about the business itself ,
how , bubbling just beneath the surface of big - town americana , there 's a sordid world of sick and depraved people who wo n't necessarily stop short of murder in order to satisfy their sick and twisted desires ?
or is it about those who can , those who are in a position to influence the making of the kinds of films sick and demented people want to see ?
i 'm not talking about snuff films , supposed " documentaries " of victims being brutalized and killed on camera .
i 'm talking about films like " 8 mm " and its director , joel schumacher .
with a recent run of big budget movies to his credit-- " batman & robin , " " a time to kill , " " batman forever , " " the client
" --schumacher certainly has that kind of influence .
is " 8 mm " something you really want to see ? probably not .
the first two - thirds of " 8 mm " unwind as a fairly conventional missing persons drama , albeit with a particularly unsavory core .
then , as it 's been threatening all along , the film explodes into violence .
and just when you think it 's finally over , schumacher tags on a ridiculous self - righteous finale that drags the whole unpleasant experience down even further . trust me .
there are better ways to waste two hours of your life .
nicolas ' " snake eyes " ' cage plays private investigator tom welles who is hired by a wealthy philadelphia widow to determine whether a reel of film found in her late husband 's safe documents a young girl 's murder .
welles goes about his assignment rather matter - of - factly , and the pieces of the puzzle fall into place rather neatly , almost as if you do n't need any specialized skills or training to do this .
welles certainly makes it look easy .
and cops , obviously , never look in toilet tanks for clues .
the deeper welles digs into his investigation the more obsessed he becomes , like george c .
scott in paul schrader 's " hardcore .
"
occasionally , a little flickering sound whirs in his head like sprockets winding through a film projector , reminding him of his unpleasant task .
there are hints that this is taking its toll on his lovely wife , played by catherine keener , who is frustrated by her husband spending all of his time in cleveland rather than in their ugly split - level home in harrisburg , pa .
"
8 mm " does n't condemn or condone its subject matter , it just exploits it .
the irony , of course , is that schumacher and " seven " scribe andrew kevin walker 's vision of life in the snuff lane is limited by what they can show in an r - rated , first - run hollywood product .
so we only see snippets of snuff , and a lot more footage of nicolas cage covering his face in horror .
later it 's the turn of joaquin phoenix ( who 's quite good and by far the film 's most interesting character as adult bookstore flunky max california ) to cover his face as the horrid thing is screened over and over again .
all this to get to the familiar yet offensive " revelation " that sexual deviants are not , indeed , monsters but everyday people like you and me .
neither super nor standard , " 8 mm " is shocking only in its banality .


that 's exactly how long the movie felt to me .
there were n't even nine laughs in nine months .
it 's a terrible mess of a movie starring a terrible mess of a man , mr . hugh grant , a huge dork .
it 's not the whole oral - sex / prostitution thing ( referring to grant , not me ) that bugs me , it 's the fact that grant is annoying .
not just adam sandler - annoying , we 're talking jim carrey - annoying .
since when do
eye flutters and nervous smiles pass for acting ?
but , on the other hand , since when do really bad slapstick ( a fistfight in the delivery room culminating in grant 's head in joan cusack 's lap -- a scene he paid $ 60 to have included in the movie ) and obscene double entendres ( robin williams , the obstetrician , tells grant 's pregnant girlfriend she has " a big pussy , " referring of course to the size of the cat hairs on her coat , but nonetheless , grant paid $ 60 to have the exchange included in the movie ) pass for comedy ?
nine months is a predictable cookie - cutter movie with
no originality in humor or plot .
hugh grant plays a successful child psychiatrist .
why a child psychologist ?
so the scriptwriters could inject the following unfunny exchange :
kid : my dad 's an asshole .
grant ( flutters eyelashes , offers a nervous smile , then responds in his annoying english accent and i - think - i - actually - have- talent attitude ) : could you possibly elaborate on that ?
kid :
my dad 's a _ huge _ asshole .
more like a hugh asshole , but that 's beside the point , which is : nine months includes too many needlessly stupid jokes that get laughs from the ten year olds in the audience while everyone else shakes his or her head in disbelief .
so , anyway , grant finds out his girlfriend is pregnant and does his usual reaction ( fluttered eyelashes , nervous smiles ) .
this paves the way for every possible pregnancy / child birth gag in the book , especially since grant 's equally annoying friend 's wife is also pregnant .
the annoying friend is played by tom arnold , who provides most of the cacophonous slapstick , none of which is funny , such as a scene where arnold beats up a costumed " arnie the dinosaur " ( you draw your own parallels on that one ) in a toy store .
the only interesting character in the movie is played by jeff goldblum , who should have hid himself away somewhere after the dreadful hideaway , as an artist with a fear of ( and simultaneous longing for ) commitment .
not even robin williams , who plays a russian doctor who has recently decided to switch from veterinary medicine to obstetrics , has much humor .
his is a one - joke character--
the old foreign - guy - who - mispronounces - english stereotype ( did someone say yakov smirnov ?
that 's my favorite vodka , by the way ) , hence the line
" now it 's time to take a look at your volvo ,
" another nasty but unamusing joke , except this one goes right over the ten year olds ' heads , while the adults simultaneously groan .
nine months is a complete failure , low on laughs and intelligence and high on loud , unfunny slapstick , failed jokes and other uninspired lunacy .
hugh grant 's sunset boulevard arrest ( please , no caught - with - his - pants - down jokes ) may bring more people into the theaters , but they certainly wo n't leave with a smile on their faces , not after 90 minutes of grant 's nervous smiles .
everything in the movie is so forced , so unauthentic that anyone with an i .
q . over 80 ( sorry , hugh ) will know
they wasted their money on an unfulfilled desire .
but at least they did n't spend 60 bucks for it .


call it a road trip for the walking wounded .
stellan skarsg ?
rd plays such a convincingly zombified drunken loser that it 's difficult to spend nearly two hours of screen time in his smelly , boozed - out presence .
yet this ever - reliable swedish actor adds depth and significance to the otherwise plodding and forgettable aberdeen ,
a sentimental and painfully mundane european drama .
playwright august strindberg built his career on families and relationships paralyzed by secrets , unable to express their longings until the hour is far too late .
that 's an accurate reflection of what aberdeen strives for , focusing on the pairing of an alcoholic father , tomas ( skarsg ? rd ) and his alienated , openly hostile yuppie daughter , kaisa ( lena headey , gossip ) .
they have n't spoken in years , and would n't even be making the long trip from norway to aberdeen , scotland by automobile if it were n't for kaisa 's mother ( charlotte rampling , under the sand ) rotting away in a hospital bed from cancer .
in a soap opera twist , mother has only a few days to live .
( only in the movies , right ? )
too blitzed to even step foot on a plane , tomas hits the open road with kaisa .
loathing each other all the while , they make periodic stops for tomas to puke on the dashboard or pass out -- whenever he is n't muttering
what a rotten kid she turned out to be .
despite his sloshed viewpoint , tomas recognizes that the apple has n't fallen very far from the tree .
kaisa gets nosebleeds from snorting coke , sabotages her personal relationships through indifference , and is unable to restrain her quick and vindictive temper .
ai n't they a pair ? unable to find true notes of unspoken familial empathy in the one - note and repetitively bitchy dialogue , screenwriters kristin amundsen and hans petter moland fabricate a series of contrivances to propel events forward -- lost money , roving street hooligans looking for drunks to kick around , nosy cops , and flat tires all figure into the schematic and convenient narrative .
by the time they reach the hospital , it 's time to unveil the secrets from a dark past that are not only simplistic devices that trivialize the father - daughter conflict , they 're also the mainstays of many a bad strindberg wannabe .
this revelation exists purely for its own sake .
aberdeen does n't know where else to go .
weak , unimaginative casting thwarts the pivotal role of kaisa .
if lena headey were a stronger actress , perhaps aberdeen could have been able to coast on the performances and moody , haunting cinematography ( rendering norway into its own pastoral ghost world -- the reference to a certain superior american indie flick intentional ) .
headey 's too busy acting , using her face and furrowed brow to convey every last twitch of insouciance .
if she were paying any attention to skarsg ?
rd , maybe she 'd figure out that doing less can reveal so much more .
it 's worthwhile to compare aberdeen to an earlier film released in 2001 , jonathan nossiter 's captivating signs & wonders .
it 's not
just because skarsg ?
rd and rampling played disturbed parental figures in both films
( they 're not bound by ceremonial wedlock in aberdeen ) .
the differences in the way their characters were presented is significant .
in aberdeen , rampling is a luminous diva , preening and static in her hospital bed .
despite skarsg ?
rd 's solid performance as tomas , his pathetic drunk is never given much of a chance to emote anything besides catatonic sorrow .
there 's genuine ferocity and sexually charged frisson during their understated confrontations in signs & wonders , allowing them to suggest a gray zone of complications that accompany torn romance and years of stifled curiosity .
nossiter 's film thoroughly explores this neurotic territory in addition to delving into the americanization of greece and the use of mysticism as an illusion to deflect pain .
if signs & wonders sometimes feels overloaded with ideas , at least it 's willing to stretch beyond what we 've come to expect from traditional drama .
aberdeen is never half so ambitious , content to sleepwalk through the rhythms and timing of other movies .
when did character driven stories stop paying attention to the complexities of real life ?
the depressing answer can be found in lawrence kasdan 's trite but occasionally useful grand canyon , where steve martin 's hollywood mogul pronounces , " all of life 's riddles are answered in the movies !
" even foreign films are taking that advice to heart .


plot : a young french boy sees his parents killed before his eyes by tim roth , oops . . .
i mean , an evil man .
he vows revenge on that man and is taught the ways of the musketeer by some old dude who used to be one himself ?
anyway , fourteen years go by and . . .
arrgh , well , you know the rest .
. .
swish - swish - zzzzzzz ! critique :
this is a pretty bad movie .
let 's see , where should i start ?
okay , first of all ,
the story is just plain boring .
it 's not original , is entirely predictable and lacks energy
.
okay , what 's next ?
acting , you say .
hmmmm , well , the main actor , justin chambers , is basically an uncharismatic version of chris o'donnell but with less range ( think about that ! ) , and mena suvari , is just plain off .
not as bad as thora birch was in dungeons & dragons , but entirely miscast , with bad deliveries , awful sequences and a piss - poor accent
that comes and goes .
now i 'm not sure if this was ms .
suvari 's fault or the director 's , but
i 've definitely seen her at a much higher level than in this film .
the only semi - saving grace actor - wise is tim roth as the irrepressible " bad guy " , but once again , it 's not something that we have n't seen before . . .
a thousand times . . .
by the same guy ! !
tim , please . . . for the love of god , beg your agent to ask the marketplace for some modern day " american roles " for you as a " nice guy " in a romantic comedy or something .
stretch , dude . . .
stretch ! !
we all know that you can do much better than this gunk .
alright ,
what else was bad in this film ?
oh yeah , the score ! yikes , how 's about taking it down a few notches there , fellas ?
this thing blares in your ear whenever it feels the need to accentuate a certain scene , but actually does little more than annoy
.
i think it 's important for the man behind the music to recognize that this film is n't a " real epic " by any stretch of the imagination .
it 's a fluffy rehashed cake - walk created by some " shrewd " studio heads who decided to take advantage of the whole " kung - fu " phenomenon in films , and test it out on an old classic .
dudes . . .
you failed all around !
( keep reading )
the editing is also pretty shoddy in this movie , the dialogue banal and stilted and the plot problems . . .
plentiful !
( why does the guy on top of the horse carriage just stand there when his opponent takes forever scampering his way back to the top ?
why do n't they just cut the mouseketeer 's rope at the top of the tower , instead of jumping down on their own chords and fighting him while hanging ?
why does n't anybody look a day older , when the sequence says " 14 years later " ?
( at least . . . change your shirt , man ! )
keep in mind that i have never strayed away from championing certain movies that are created simply for the sake of a " fun time " , but this flick just did n't cut it for me .
it was boring for stretches ,
the acting was atrocious at times
( the " romantic " scene between suvari and chambers next to the lake reminded me of plays in high school which made you cringe ) , there was little reason to care for anyone and since when were the musketeers fat ?
i will give the movie this much , and that is that its main reason for being ( its " raison - d'etre " , as the french would say ) , its fight sequences , do come through despite the lack of their numbers in the film .
i was hoping that the movie would be packed with cool stuntwork as promoted in its trailer , but what you see there , are essentially the snippets from the two major ( and cool ) swashbuckling sequences from the film .
the first comes right at the beginning of the movie , while the other essentially finishes the film off , hanging from the tower and juggling off ladders .
the ladder sequence itself is a definite keeper but unfortunately the rest of the movie is just regurgitated crap .
and can anyone please tell me how catherine deneuve got her name placed at the top of this film 's credits ?
hullo ?
the film is called the musketeer and stars a dude name justin chambers .
deneuve is barely in this movie !
ugh , just another small thing that annoyed me about this trash .
now say it together , gang : " all for one , and one for all . . .
we vow to stay away from it all ! !
" thank me later .
where 's joblo coming from ?
a knight 's tale ( 7/10 ) - american outlaws ( 5/10 ) - crouching tiger , hidden dragon ( 7/10 ) - the matrix ( 8/10 ) - the replacement killers ( 6/10 )
- romeo must die ( 3/10 )
- shanghai noon ( 6/10 )


from django import forms
from django.core.exceptions import ValidationError
from django.core.validators import validate_slug
from django.db import models
from django.utils import simplejson as json
from django.utils.text import capfirst
from django.utils.translation import ugettext_lazy as _

from philo.forms.fields import JSONFormField
from philo.utils.registry import RegistryIterator
from philo.validators import TemplateValidator, json_validator
#from philo.models.fields.entities import *

class TemplateField(models.TextField):
	"""A :class:`TextField` which is validated with a :class:`.TemplateValidator`. ``allow``, ``disallow``, and ``secure`` will be passed into the validator's construction."""
	def __init__(self, allow=None, disallow=None, secure=True, *args, **kwargs):
		super(TemplateField, self).__init__(*args, **kwargs)
		self.validators.append(TemplateValidator(allow, disallow, secure))

class JSONDescriptor(object):
	def __init__(self, field):
		self.field = field

	def __get__(self, instance, owner):
		if instance is None:
			raise AttributeError # ?

		if self.field.name not in instance.__dict__:
			json_string = getattr(instance, self.field.attname)
			instance.__dict__[self.field.name] = json.loads(json_string)

		return instance.__dict__[self.field.name]

	def __set__(self, instance, value):
		instance.__dict__[self.field.name] = value
		setattr(instance, self.field.attname, json.dumps(value))

	def __delete__(self, instance):
		del(instance.__dict__[self.field.name])
		setattr(instance, self.field.attname, json.dumps(None))

class JSONField(models.TextField):
	"""A :class:`TextField` which stores its value on the model instance as a python object and stores its value in the database as JSON. Validated with :func:`.json_validator`."""
	default_validators = [json_validator]

	def get_attname(self):
		return "%s_json" % self.name

	def contribute_to_class(self, cls, name):
		super(JSONField, self).contribute_to_class(cls, name)
		setattr(cls, name, JSONDescriptor(self))
		models.signals.pre_init.connect(self.fix_init_kwarg, sender=cls)

	def fix_init_kwarg(self, sender, args, kwargs, **signal_kwargs):
		# Anything passed in as self.name is assumed to come from a serializer and
		# will be treated as a json string.
		if self.name in kwargs:
			value = kwargs.pop(self.name)

			# Hack to handle the xml serializer's handling of "null"
			if value is None:
				value = 'null'

			kwargs[self.attname] = value

	def formfield(self, *args, **kwargs):
		kwargs["form_class"] = JSONFormField
		return super(JSONField, self).formfield(*args, **kwargs)

class SlugMultipleChoiceField(models.Field):
	"""Stores a selection of multiple items with unique slugs in the form of a comma-separated list. Also knows how to correctly handle :class:`RegistryIterator`\ s passed in as choices."""
	__metaclass__ = models.SubfieldBase
	description = _("Comma-separated slug field")

	def get_internal_type(self):
		return "TextField"

	def to_python(self, value):
		if not value:
			return []

		if isinstance(value, list):
			return value

		return value.split(',')

	def get_prep_value(self, value):
		return ','.join(value)

	def formfield(self, **kwargs):
		# This is necessary because django hard-codes TypedChoiceField for things with choices.
		defaults = {
			'widget': forms.CheckboxSelectMultiple,
			'choices': self.get_choices(include_blank=False),
			'label': capfirst(self.verbose_name),
			'required': not self.blank,
			'help_text': self.help_text
		}
		if self.has_default():
			if callable(self.default):
				defaults['initial'] = self.default
				defaults['show_hidden_initial'] = True
			else:
				defaults['initial'] = self.get_default()

		for k in kwargs.keys():
			if k not in ('coerce', 'empty_value', 'choices', 'required',
						 'widget', 'label', 'initial', 'help_text',
						 'error_messages', 'show_hidden_initial'):
				del kwargs[k]

		defaults.update(kwargs)
		form_class = forms.TypedMultipleChoiceField
		return form_class(**defaults)

	def validate(self, value, model_instance):
		invalid_values = []
		for val in value:
			try:
				validate_slug(val)
			except ValidationError:
				invalid_values.append(val)

		if invalid_values:
			# should really make a custom message.
			raise ValidationError(self.error_messages['invalid_choice'] % invalid_values)

	def _get_choices(self):
		if isinstance(self._choices, RegistryIterator):
			return self._choices.copy()
		elif hasattr(self._choices, 'next'):
			choices, self._choices = itertools.tee(self._choices)
			return choices
		else:
			return self._choices
	choices = property(_get_choices)

try:
	from south.modelsinspector import add_introspection_rules
except ImportError:
	pass
else:
	add_introspection_rules([], ["^philo\.models\.fields\.SlugMultipleChoiceField"])
	add_introspection_rules([], ["^philo\.models\.fields\.TemplateField"])
	add_introspection_rules([], ["^philo\.models\.fields\.JSONField"])


import hashlib
import json
import logging
import os
import subprocess
import sys
import time
from collections import defaultdict

from shutil import copy
from shutil import copyfile
from shutil import copystat
from shutil import copytree
from tempfile import mkdtemp

import boto3
import botocore
import yaml
import sys

from .helpers import archive
from .helpers import get_environment_variable_value
from .helpers import LambdaContext
from .helpers import mkdir
from .helpers import read
from .helpers import timestamp

ARN_PREFIXES = {
    "cn-north-1": "aws-cn",
    "cn-northwest-1": "aws-cn",
    "us-gov-west-1": "aws-us-gov",
}

log = logging.getLogger(__name__)

def load_source(module_name, module_path):
    """Loads a python module from the path of the corresponding file."""

    if sys.version_info[0] == 3 and sys.version_info[1] >= 5:
        import importlib.util
        spec = importlib.util.spec_from_file_location(module_name, module_path)
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
    elif sys.version_info[0] == 3 and sys.version_info[1] < 5:
        import importlib.machinery
        loader = importlib.machinery.SourceFileLoader(module_name, module_path)
        module = loader.load_module()
    return module

def cleanup_old_versions(
    src, keep_last_versions, config_file="config.yaml", profile_name=None,
):
    """Deletes old deployed versions of the function in AWS Lambda.

    Won't delete $Latest and any aliased version

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param int keep_last_versions:
        The number of recent versions to keep and not delete
    """
    if keep_last_versions <= 0:
        print("Won't delete all versions. Please do this manually")
    else:
        path_to_config_file = os.path.join(src, config_file)
        cfg = read_cfg(path_to_config_file, profile_name)

        profile_name = cfg.get("profile")
        aws_access_key_id = cfg.get("aws_access_key_id")
        aws_secret_access_key = cfg.get("aws_secret_access_key")

        client = get_client(
            "lambda",
            profile_name,
            aws_access_key_id,
            aws_secret_access_key,
            cfg.get("region"),
        )

        response = client.list_versions_by_function(
            FunctionName=cfg.get("function_name"),
        )
        versions = response.get("Versions")
        if len(response.get("Versions")) < keep_last_versions:
            print("Nothing to delete. (Too few versions published)")
        else:
            version_numbers = [
                elem.get("Version") for elem in versions[1:-keep_last_versions]
            ]
            for version_number in version_numbers:
                try:
                    client.delete_function(
                        FunctionName=cfg.get("function_name"),
                        Qualifier=version_number,
                    )
                except botocore.exceptions.ClientError as e:
                    print(f"Skipping Version {version_number}: {e}")

def deploy(
    src,
    requirements=None,
    local_package=None,
    config_file="config.yaml",
    profile_name=None,
    preserve_vpc=False,
):
    """Deploys a new function to AWS Lambda.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)
    """
    # Load and parse the config file.
    path_to_config_file = os.path.join(src, config_file)
    cfg = read_cfg(path_to_config_file, profile_name)

    # Copy all the pip dependencies required to run your code into a temporary
    # folder then add the handler file in the root of this directory.
    # Zip the contents of this folder into a single file and output to the dist
    # directory.
    path_to_zip_file = build(
        src,
        config_file=config_file,
        requirements=requirements,
        local_package=local_package,
    )

    existing_config = get_function_config(cfg)
    if existing_config:
        update_function(
            cfg, path_to_zip_file, existing_config, preserve_vpc=preserve_vpc
        )
    else:
        create_function(cfg, path_to_zip_file)

def deploy_s3(
    src,
    requirements=None,
    local_package=None,
    config_file="config.yaml",
    profile_name=None,
    preserve_vpc=False,
):
    """Deploys a new function via AWS S3.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)
    """
    # Load and parse the config file.
    path_to_config_file = os.path.join(src, config_file)
    cfg = read_cfg(path_to_config_file, profile_name)

    # Copy all the pip dependencies required to run your code into a temporary
    # folder then add the handler file in the root of this directory.
    # Zip the contents of this folder into a single file and output to the dist
    # directory.
    path_to_zip_file = build(
        src,
        config_file=config_file,
        requirements=requirements,
        local_package=local_package,
    )

    use_s3 = True
    s3_file = upload_s3(cfg, path_to_zip_file, use_s3)
    existing_config = get_function_config(cfg)
    if existing_config:
        update_function(
            cfg,
            path_to_zip_file,
            existing_config,
            use_s3=use_s3,
            s3_file=s3_file,
            preserve_vpc=preserve_vpc,
        )
    else:
        create_function(cfg, path_to_zip_file, use_s3=use_s3, s3_file=s3_file)

def upload(
    src,
    requirements=None,
    local_package=None,
    config_file="config.yaml",
    profile_name=None,
):
    """Uploads a new function to AWS S3.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)
    """
    # Load and parse the config file.
    path_to_config_file = os.path.join(src, config_file)
    cfg = read_cfg(path_to_config_file, profile_name)

    # Copy all the pip dependencies required to run your code into a temporary
    # folder then add the handler file in the root of this directory.
    # Zip the contents of this folder into a single file and output to the dist
    # directory.
    path_to_zip_file = build(
        src,
        config_file=config_file,
        requirements=requirements,
        local_package=local_package,
    )

    upload_s3(cfg, path_to_zip_file)

def invoke(
    src,
    event_file="event.json",
    config_file="config.yaml",
    profile_name=None,
    verbose=False,
):
    """Simulates a call to your function.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str alt_event:
        An optional argument to override which event file to use.
    :param bool verbose:
        Whether to print out verbose details.
    """
    # Load and parse the config file.
    path_to_config_file = os.path.join(src, config_file)
    cfg = read_cfg(path_to_config_file, profile_name)

    # Set AWS_PROFILE environment variable based on `--profile` option.
    if profile_name:
        os.environ["AWS_PROFILE"] = profile_name

    # Load environment variables from the config file into the actual
    # environment.
    env_vars = cfg.get("environment_variables")
    if env_vars:
        for key, value in env_vars.items():
            os.environ[key] = get_environment_variable_value(value)

    # Load and parse event file.
    path_to_event_file = os.path.join(src, event_file)
    event = read(path_to_event_file, loader=json.loads)

    # Tweak to allow module to import local modules
    try:
        sys.path.index(src)
    except ValueError:
        sys.path.append(src)

    handler = cfg.get("handler")
    # Inspect the handler string (<module>.<function name>) and translate it
    # into a function we can execute.
    fn = get_callable_handler_function(src, handler)

    timeout = cfg.get("timeout")
    if timeout:
        context = LambdaContext(cfg.get("function_name"), timeout)
    else:
        context = LambdaContext(cfg.get("function_name"))

    start = time.time()
    results = fn(event, context)
    end = time.time()

    print("{0}".format(results))
    if verbose:
        print(
            "\nexecution time: {:.8f}s\nfunction execution "
            "timeout: {:2}s".format(end - start, cfg.get("timeout", 15))
        )

def init(src, minimal=False):
    """Copies template files to a given directory.

    :param str src:
        The path to output the template lambda project files.
    :param bool minimal:
        Minimal possible template files (excludes event.json).
    """

    templates_path = os.path.join(
        os.path.dirname(os.path.abspath(__file__)), "project_templates",
    )
    for filename in os.listdir(templates_path):
        if (minimal and filename == "event.json") or filename.endswith(".pyc"):
            continue
        dest_path = os.path.join(templates_path, filename)

        if not os.path.isdir(dest_path):
            copy(dest_path, src)

def build(
    src,
    requirements=None,
    local_package=None,
    config_file="config.yaml",
    profile_name=None,
):
    """Builds the file bundle.

    :param str src:
       The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)
    """
    # Load and parse the config file.
    path_to_config_file = os.path.join(src, config_file)
    cfg = read_cfg(path_to_config_file, profile_name)

    # Get the absolute path to the output directory and create it if it doesn't
    # already exist.
    dist_directory = cfg.get("dist_directory", "dist")
    path_to_dist = os.path.join(src, dist_directory)
    mkdir(path_to_dist)

    # Combine the name of the Lambda function with the current timestamp to use
    # for the output filename.
    function_name = cfg.get("function_name")
    output_filename = "{0}-{1}.zip".format(timestamp(), function_name)

    path_to_temp = mkdtemp(prefix="aws-lambda")
    pip_install_to_target(
        path_to_temp, requirements=requirements, local_package=local_package,
    )

    # Hack for Zope.
    if "zope" in os.listdir(path_to_temp):
        print(
            "Zope packages detected; fixing Zope package paths to "
            "make them importable.",
        )
        # Touch.
        with open(os.path.join(path_to_temp, "zope/__init__.py"), "wb"):
            pass

    # Gracefully handle whether ".zip" was included in the filename or not.
    output_filename = (
        "{0}.zip".format(output_filename)
        if not output_filename.endswith(".zip")
        else output_filename
    )

    # Allow definition of source code directories we want to build into our
    # zipped package.
    build_config = defaultdict(**cfg.get("build", {}))
    build_source_directories = build_config.get("source_directories", "")
    build_source_directories = (
        build_source_directories
        if build_source_directories is not None
        else ""
    )
    source_directories = [
        d.strip() for d in build_source_directories.split(",")
    ]

    files = []
    for filename in os.listdir(src):
        if os.path.isfile(filename):
            if filename == ".DS_Store":
                continue
            if filename == config_file:
                continue
            print("Bundling: %r" % filename)
            files.append(os.path.join(src, filename))
        elif os.path.isdir(filename) and filename in source_directories:
            print("Bundling directory: %r" % filename)
            files.append(os.path.join(src, filename))

    # "cd" into `temp_path` directory.
    os.chdir(path_to_temp)
    for f in files:
        if os.path.isfile(f):
            _, filename = os.path.split(f)

            # Copy handler file into root of the packages folder.
            copyfile(f, os.path.join(path_to_temp, filename))
            copystat(f, os.path.join(path_to_temp, filename))
        elif os.path.isdir(f):
            src_path_length = len(src) + 1
            destination_folder = os.path.join(
                path_to_temp, f[src_path_length:]
            )
            copytree(f, destination_folder)

    # Zip them together into a single file.
    # TODO: Delete temp directory created once the archive has been compiled.
    path_to_zip_file = archive("./", path_to_dist, output_filename)
    return path_to_zip_file

def get_callable_handler_function(src, handler):
    """Translate a string of the form "module.function" into a callable
    function.

    :param str src:
      The path to your Lambda project containing a valid handler file.
    :param str handler:
      A dot delimited string representing the `<module>.<function name>`.
    """

    # "cd" into `src` directory.
    os.chdir(src)

    module_name, function_name = handler.split(".")
    filename = get_handler_filename(handler)

    path_to_module_file = os.path.join(src, filename)
    module = load_source(module_name, path_to_module_file)
    return getattr(module, function_name)

def get_handler_filename(handler):
    """Shortcut to get the filename from the handler string.

    :param str handler:
      A dot delimited string representing the `<module>.<function name>`.
    """
    module_name, _ = handler.split(".")
    return "{0}.py".format(module_name)

def _install_packages(path, packages):
    """Install all packages listed to the target directory.

    Ignores any package that includes Python itself and python-lambda as well
    since its only needed for deploying and not running the code

    :param str path:
        Path to copy installed pip packages to.
    :param list packages:
        A list of packages to be installed via pip.
    """

    def _filter_blacklist(package):
        blacklist = ["-i", "#", "Python==", "python-lambda=="]
        return all(package.startswith(entry) is False for entry in blacklist)

    filtered_packages = filter(_filter_blacklist, packages)
    for package in filtered_packages:
        if package.startswith("-e "):
            package = package.replace("-e ", "")

        print("Installing {package}".format(package=package))
        subprocess.check_call(
            [
                sys.executable,
                "-m",
                "pip",
                "install",
                package,
                "-t",
                path,
                "--ignore-installed",
            ]
        )
    print(
        "Install directory contents are now: {directory}".format(
            directory=os.listdir(path)
        )
    )

def pip_install_to_target(path, requirements=None, local_package=None):
    """For a given active virtualenv, gather all installed pip packages then
    copy (re-install) them to the path provided.

    :param str path:
        Path to copy installed pip packages to.
    :param str requirements:
        If set, only the packages in the supplied requirements file are
        installed.
        If not set then installs all packages found via pip freeze.
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)
    """
    packages = []
    if not requirements:
        print("Gathering pip packages")
        pkgStr = subprocess.check_output(
            [sys.executable, "-m", "pip", "freeze"]
        )
        packages.extend(pkgStr.decode("utf-8").splitlines())
    else:
        if os.path.exists(requirements):
            print("Gathering requirement packages")
            data = read(requirements)
            packages.extend(data.splitlines())

    if not packages:
        print("No dependency packages installed!")

    if local_package is not None:
        if not isinstance(local_package, (list, tuple)):
            local_package = [local_package]
        for l_package in local_package:
            packages.append(l_package)
    _install_packages(path, packages)

def get_role_name(region, account_id, role):
    """Shortcut to insert the `account_id` and `role` into the iam string."""
    prefix = ARN_PREFIXES.get(region, "aws")
    return "arn:{0}:iam::{1}:role/{2}".format(prefix, account_id, role)

def get_account_id(
    profile_name, aws_access_key_id, aws_secret_access_key, region=None,
):
    """Query STS for a users' account_id"""
    client = get_client(
        "sts", profile_name, aws_access_key_id, aws_secret_access_key, region,
    )
    return client.get_caller_identity().get("Account")

def get_client(
    client,
    profile_name,
    aws_access_key_id,
    aws_secret_access_key,
    region=None,
):
    """Shortcut for getting an initialized instance of the boto3 client."""

    boto3.setup_default_session(
        profile_name=profile_name,
        aws_access_key_id=aws_access_key_id,
        aws_secret_access_key=aws_secret_access_key,
        region_name=region,
    )
    return boto3.client(client)

def create_function(cfg, path_to_zip_file, use_s3=False, s3_file=None):
    """Register and upload a function to AWS Lambda."""

    print("Creating your new Lambda function")
    byte_stream = read(path_to_zip_file, binary_file=True)
    profile_name = cfg.get("profile")
    aws_access_key_id = cfg.get("aws_access_key_id")
    aws_secret_access_key = cfg.get("aws_secret_access_key")

    account_id = get_account_id(
        profile_name,
        aws_access_key_id,
        aws_secret_access_key,
        cfg.get("region",),
    )
    role = get_role_name(
        cfg.get("region"),
        account_id,
        cfg.get("role", "lambda_basic_execution"),
    )

    client = get_client(
        "lambda",
        profile_name,
        aws_access_key_id,
        aws_secret_access_key,
        cfg.get("region"),
    )

    # Do we prefer development variable over config?
    buck_name = os.environ.get("S3_BUCKET_NAME") or cfg.get("bucket_name")
    func_name = os.environ.get("LAMBDA_FUNCTION_NAME") or cfg.get(
        "function_name"
    )
    print("Creating lambda function with name: {}".format(func_name))

    if use_s3:
        kwargs = {
            "FunctionName": func_name,
            "Runtime": cfg.get("runtime", "python2.7"),
            "Role": role,
            "Handler": cfg.get("handler"),
            "Code": {
                "S3Bucket": "{}".format(buck_name),
                "S3Key": "{}".format(s3_file),
            },
            "Description": cfg.get("description", ""),
            "Timeout": cfg.get("timeout", 15),
            "MemorySize": cfg.get("memory_size", 512),
            "VpcConfig": {
                "SubnetIds": cfg.get("subnet_ids", []),
                "SecurityGroupIds": cfg.get("security_group_ids", []),
            },
            "Publish": True,
        }
    else:
        kwargs = {
            "FunctionName": func_name,
            "Runtime": cfg.get("runtime", "python2.7"),
            "Role": role,
            "Handler": cfg.get("handler"),
            "Code": {"ZipFile": byte_stream},
            "Description": cfg.get("description", ""),
            "Timeout": cfg.get("timeout", 15),
            "MemorySize": cfg.get("memory_size", 512),
            "VpcConfig": {
                "SubnetIds": cfg.get("subnet_ids", []),
                "SecurityGroupIds": cfg.get("security_group_ids", []),
            },
            "Publish": True,
        }

    if "tags" in cfg:
        kwargs.update(
            Tags={key: str(value) for key, value in cfg.get("tags").items()}
        )

    if "environment_variables" in cfg:
        kwargs.update(
            Environment={
                "Variables": {
                    key: get_environment_variable_value(value)
                    for key, value in cfg.get("environment_variables").items()
                },
            },
        )

    client.create_function(**kwargs)

    concurrency = get_concurrency(cfg)
    if concurrency > 0:
        client.put_function_concurrency(
            FunctionName=func_name, ReservedConcurrentExecutions=concurrency
        )

def update_function(
    cfg,
    path_to_zip_file,
    existing_cfg,
    use_s3=False,
    s3_file=None,
    preserve_vpc=False,
):
    """Updates the code of an existing Lambda function"""

    print("Updating your Lambda function")
    byte_stream = read(path_to_zip_file, binary_file=True)
    profile_name = cfg.get("profile")
    aws_access_key_id = cfg.get("aws_access_key_id")
    aws_secret_access_key = cfg.get("aws_secret_access_key")

    account_id = get_account_id(
        profile_name,
        aws_access_key_id,
        aws_secret_access_key,
        cfg.get("region",),
    )
    role = get_role_name(
        cfg.get("region"),
        account_id,
        cfg.get("role", "lambda_basic_execution"),
    )

    client = get_client(
        "lambda",
        profile_name,
        aws_access_key_id,
        aws_secret_access_key,
        cfg.get("region"),
    )

    # Do we prefer development variable over config?
    buck_name = os.environ.get("S3_BUCKET_NAME") or cfg.get("bucket_name")

    if use_s3:
        client.update_function_code(
            FunctionName=cfg.get("function_name"),
            S3Bucket="{}".format(buck_name),
            S3Key="{}".format(s3_file),
            Publish=True,
        )
    else:
        client.update_function_code(
            FunctionName=cfg.get("function_name"),
            ZipFile=byte_stream,
            Publish=True,
        )

    kwargs = {
        "FunctionName": cfg.get("function_name"),
        "Role": role,
        "Runtime": cfg.get("runtime"),
        "Handler": cfg.get("handler"),
        "Description": cfg.get("description", ""),
        "Timeout": cfg.get("timeout", 15),
        "MemorySize": cfg.get("memory_size", 512),
    }

    if preserve_vpc:
        kwargs["VpcConfig"] = existing_cfg.get("Configuration", {}).get(
            "VpcConfig"
        )
        if kwargs["VpcConfig"] is None:
            kwargs["VpcConfig"] = {
                "SubnetIds": cfg.get("subnet_ids", []),
                "SecurityGroupIds": cfg.get("security_group_ids", []),
            }
        else:
            del kwargs["VpcConfig"]["VpcId"]
    else:
        kwargs["VpcConfig"] = {
            "SubnetIds": cfg.get("subnet_ids", []),
            "SecurityGroupIds": cfg.get("security_group_ids", []),
        }

    if "environment_variables" in cfg:
        kwargs.update(
            Environment={
                "Variables": {
                    key: str(get_environment_variable_value(value))
                    for key, value in cfg.get("environment_variables").items()
                },
            },
        )

    ret = client.update_function_configuration(**kwargs)

    concurrency = get_concurrency(cfg)
    if concurrency > 0:
        client.put_function_concurrency(
            FunctionName=cfg.get("function_name"),
            ReservedConcurrentExecutions=concurrency,
        )
    elif "Concurrency" in existing_cfg:
        client.delete_function_concurrency(
            FunctionName=cfg.get("function_name")
        )

    if "tags" in cfg:
        tags = {key: str(value) for key, value in cfg.get("tags").items()}
        if tags != existing_cfg.get("Tags"):
            if existing_cfg.get("Tags"):
                client.untag_resource(
                    Resource=ret["FunctionArn"],
                    TagKeys=list(existing_cfg["Tags"].keys()),
                )
            client.tag_resource(Resource=ret["FunctionArn"], Tags=tags)

def upload_s3(cfg, path_to_zip_file, *use_s3):
    """Upload a function to AWS S3."""

    print("Uploading your new Lambda function")
    profile_name = cfg.get("profile")
    aws_access_key_id = cfg.get("aws_access_key_id")
    aws_secret_access_key = cfg.get("aws_secret_access_key")
    client = get_client(
        "s3",
        profile_name,
        aws_access_key_id,
        aws_secret_access_key,
        cfg.get("region"),
    )
    byte_stream = b""
    with open(path_to_zip_file, mode="rb") as fh:
        byte_stream = fh.read()
    s3_key_prefix = cfg.get("s3_key_prefix", "/dist")
    checksum = hashlib.new("md5", byte_stream).hexdigest()
    timestamp = str(time.time())
    filename = "{prefix}{checksum}-{ts}.zip".format(
        prefix=s3_key_prefix, checksum=checksum, ts=timestamp,
    )

    # Do we prefer development variable over config?
    buck_name = os.environ.get("S3_BUCKET_NAME") or cfg.get("bucket_name")
    func_name = os.environ.get("LAMBDA_FUNCTION_NAME") or cfg.get(
        "function_name"
    )
    kwargs = {
        "Bucket": "{}".format(buck_name),
        "Key": "{}".format(filename),
        "Body": byte_stream,
    }

    client.put_object(**kwargs)
    print("Finished uploading {} to S3 bucket {}".format(func_name, buck_name))
    if use_s3:
        return filename

def get_function_config(cfg):
    """Check whether a function exists or not and return its config"""

    function_name = cfg.get("function_name")
    profile_name = cfg.get("profile")
    aws_access_key_id = cfg.get("aws_access_key_id")
    aws_secret_access_key = cfg.get("aws_secret_access_key")
    client = get_client(
        "lambda",
        profile_name,
        aws_access_key_id,
        aws_secret_access_key,
        cfg.get("region"),
    )

    try:
        return client.get_function(FunctionName=function_name)
    except client.exceptions.ResourceNotFoundException as e:
        if "Function not found" in str(e):
            return False

def get_concurrency(cfg):
    """Return the Reserved Concurrent Executions if present in the config"""
    concurrency = int(cfg.get("concurrency", 0))
    return max(0, concurrency)

def read_cfg(path_to_config_file, profile_name):
    cfg = read(path_to_config_file, loader=yaml.full_load)
    if profile_name is not None:
        cfg["profile"] = profile_name
    elif "AWS_PROFILE" in os.environ:
        cfg["profile"] = os.environ["AWS_PROFILE"]
    return cfg