Drive By (The Frank & Ernest Files, #1)

Meet Ernie Campanella, the poster boy for all that was wrong with the notorious Philadelphia Police Department in the 1970s. A boarder-line alcoholic, Campanella regards bigotry and brutality as standard operating procedure. Like many racists of his day, he can look kindly upon individual black people he has met and gotten to know, but still carries a strong measure of contempt for the race in general. On the other hand, he is a genius.
Ernie's closest friend and former police academy roommate is Frank Mueller, a resourceful and ambitious cop who quickly gets promoted to Detective Third Class. The price he must pay for that promotion is to leave the orderly middle-class neighborhood where he, Ernie and their friend Greg Martin patrolled, and transfer to the most dangerous area of the city. Oddly enough, the safe, respectable neighborhood carries the unglamorous name of Fishtown, while the dangerous ghetto is called Strawberry Mansion. Go figure.
Shortly after Detective Mueller's transfer, a drive-by shooting, seemingly between rival drug dealers, takes place. The shooters are believed to be from a Jamaican gang. In the process, they kill one member of the rival gang, wound another and kill an innocent little girl who had the bad luck to be in the path of the shooters.
Because of the child's death, the city is calling for blood. Frank, to whom the case has been assigned, is under major pressure to build a case against the Jamaicans, and soon. In spite of the prevailing attitude, he thinks there may be more to this crime than meets the eye. Maybe his genius friend in Fishtown can help--that is if Frank can catch him in a sober moment.


What Is Your Dangerous Idea? : Today's Leading Thinkers on the Unthinkable

From Copernicus to Darwin, to current-day thinkers, scientists have always promoted theories and unveiled discoveries that challenge everything society holds dear; ideas with both positive and dire consequences. Many thoughts that resonate today are dangerous not because they are assumed to be false, but because they might turn out to be true.
What do the world's leading scientists and thinkers consider to be their most dangerous idea? Through the leading online forum Edge(www.edge.org), the call went out, and this compelling and easily digestible volume collects the answers. From using medication to permanently alter our personalities to contemplating a universe in which we are utterly alone, to the idea that the universe might be fundamentally inexplicable, What Is Your Dangerous Idea?takes an unflinching look at the daring, breathtaking, sometimes terrifying thoughts that could forever alter our world and the way we live in it.


A Royal Vow of Convenience

The whirr of helicopter blades announcing the return of her new boss sends Sophie scurrying back to the kitchen. Little does she know that fleeing from royal scrutiny will place her under the devastatingly searing gaze of notorious billionaire Rafe Carter.
Having resisted his cook for days, a secret midnight swim sends Rafe's senses into overdrive and he can't resist a taste of the forbidden. But the press descend when Sophie's identity is shockingly revealed, and Rafe is duty-bound to rescue the beautiful princess...with a convenient vow.


The Owl Who Was Afraid of the Dark

Plop, the Baby Barn Owl, is like every Barn Owl there ever was, except for one thing--he is afraid of the dark. "Dark is nasty" he says and so he won't go hunting with his parents. Mrs. Barn Owl sends him down from his nest-hole to ask about the dark and he meets a little boy waiting for the fireworks to begin, an old lady, a scout out camping, a girl who tells him about Father Christmas, a man with a telescope, and a black cat who takes him exploring. He realizes that through these encounters that dark is super after all.


The Rune that Binds

"When evil is beautiful, seductive, and wears the faces of the people you love, how do you fight it?"
Three thousand years after the Virtues abandoned the world to their brother, Malasephus, mankind has erected temples to the fallen Virtue,who grants power to the ambitious, glory to the vain, and wealth to the greedy; and to those who oppose him, a brutal death and then endless servitude as one of the nohetka.
Few oppose Malasephus, who is worshiped everywhere as a god. When Jan Unwin's family is brutally murdered by the nohetka, he swears an oath to avenge their deaths. Joining the nomadic Tair, Jan learns the art of the warrior, and the hidden secrets of rune magic, but his thirst for vengeance threatens not only his soul, but the entire world, and before he is finished, he will be changed in ways he cannot imagine.


"Unquestionably, the U.N. Intergovernmental Panel on Climate Change (IPCC) was formed to build the scientific case for humanity being the primary cause of global warming.

In it, the IUGG concurs with the "comprehensive and widely accepted and endorsed scientific assessments carried out by the Intergovernmental Panel on Climate Change and regional and national bodies, which have firmly established, on the basis of scientific evidence, that human activities are the primary cause of recent climate change".


Tony Abbott the Opposition must respect the Government's mandate to overturn the carbon tax.

On the first day of the new Parliament, Abbott introduced legislation into Parliament to repeal the Carbon Tax, and commenced Operation Sovereign Borders, the Coalition's policy to stop illegal maritime arrivals, which received strong public support.


[Wind energy] is a more expensive way of producing energy than the alternative.

Costs of production from coal fired plants built in "the 1970s and 1980s" are cheaper than renewable energy sources because of depreciation.


A cold day in Chicago in winter has nothing to do with the trend of global warming.

In it was the prediction that on our current course the planet will warm a disastrous seven degrees Fahrenheit (or about 3.9 degrees Celsius) by the end of this century.


Context: The final major evolution of the steam engine design was the use of steam turbines starting in the late part of the 19th century. Steam turbines are generally more efficient than reciprocating piston type steam engines (for outputs above several hundred horsepower), have fewer moving parts, and provide rotary power directly instead of through a connecting rod system or similar means. Steam turbines virtually replaced reciprocating engines in electricity generating stations early in the 20th century, where their efficiency, higher speed appropriate to generator service, and smooth rotation were advantages. Today most electric power is provided by steam turbines. In the United States 90% of the electric power is produced in this way using a variety of heat sources. Steam turbines were extensively applied for propulsion of large ships throughout most of the 20th century.

Question: What used the things more efficient than reciprocating piston type steam engines?


Context: Luther's rediscovery of "Christ and His salvation" was the first of two points that became the foundation for the Reformation. His railing against the sale of indulgences was based on it.

Question: What else was based upon the first of two points that became the foundation for the Reformation?


Context: Some chloroplasts contain a structure called the chloroplast peripheral reticulum. It is often found in the chloroplasts of C4 plants, though it has also been found in some C3 angiosperms, and even some gymnosperms. The chloroplast peripheral reticulum consists of a maze of membranous tubes and vesicles continuous with the inner chloroplast membrane that extends into the internal stromal fluid of the chloroplast. Its purpose is thought to be to increase the chloroplast's surface area for cross-membrane transport between its stroma and the cell cytoplasm. The small vesicles sometimes observed may serve as transport vesicles to shuttle stuff between the thylakoids and intermembrane space.

Question: What feature of the chloroplast is the chloroplast peripheral reticulum connected to?


Context: Two Block I CSMs were launched from LC-34 on suborbital flights in 1966 with the Saturn IB. The first, AS-201 launched on February 26, reached an altitude of 265.7 nautical miles (492.1 km) and splashed down 4,577 nautical miles (8,477 km) downrange in the Atlantic ocean. The second, AS-202 on August 25, reached 617.1 nautical miles (1,142.9 km) altitude and was recovered 13,900 nautical miles (25,700 km) downrange in the Pacific ocean. These flights validated the Service Module engine and the Command Module heat shield.

Question: Which Block I CSM was not launched in August?


Context: In addition, there are $2 million worth of other ancillary events, including a week-long event at the Santa Clara Convention Center, a beer, wine and food festival at Bellomy Field at Santa Clara University, and a pep rally. A professional fundraiser will aid in finding business sponsors and individual donors, but still may need the city council to help fund the event. Additional funding will be provided by the city council, which has announced plans to set aside seed funding for the event.

Question: what is the second source of sponsorship given besides businesses and corporations


Context: Throughout its history, the United Methodist Church has placed great emphasis on the importance of education. As such, the United Methodist Church established and is affiliated with around one hundred colleges and universities in the United States, including Syracuse University, Boston University, Emory University, Duke University, Drew University, University of Denver, University of Evansville, and Southern Methodist University. Most are members of the International Association of Methodist-related Schools, Colleges, and Universities. The church operates three hundred sixty schools and institutions overseas.

Question: What organization might have started the association of schools?


1928 in association football

The following are the football ( soccer ) events of the year 1928 throughout the world .

Question: where are the world soccer events from


1928 in association football

Question: what year was football


1928 in association football

Question: when did the world football championship begin


1986 NBA Finals

The 1986 NBA Finals was the championship round of the 1985 -- 86 NBA season . It pitted the Eastern Conference champion Boston Celtics against the Western Conference champion Houston Rockets , in a rematch of the 1981 Finals ( only Allen Leavell and Robert Reid remained from the Rockets ' 1981 team ) . The Celtics defeated the Rockets four games to two to win their 16th NBA championship . The championship would be the Celtics ' last until the 2008 NBA Finals . Larry Bird was named the Finals MVP .   On another note , this series marked the first time the `` NBA Finals '' branding was officially used , as they dropped the `` NBA World Championship Series '' branding which had been in use since the beginning of the league , though it had been unofficially called the `` NBA Finals '' for years .   Until the 2011 series , this was the last time the NBA Finals had started before June . Since game three , all NBA Finals games have been played in June . Starting with the following year , the NBA Finals would be held exclusively in the month of June . It was also the last NBA Finals series to schedule a game on a Monday until 1999 and also the last NBA Finals game to be played on Memorial Day .   CBS Sports used Dick Stockton and Tom Heinsohn as the play-by-play man and color commentator respectively . Meanwhile , Brent Musburger was the host and Pat O'Brien ( the Rockets ' sideline ) and Lesley Visser ( the Celtics ' sideline ) were the sideline reporters .

Question: when is nba finals aired?


1986 NBA Finals

Question: when does nba finals start


1986 NBA Finals

Question: what year did the nba finals begin


1901 Villanova Wildcats football team

The 1901 Villanova Wildcats football team represented the Villanova University during the 1901 college football season . The Wildcats team captain was John J. Egan .

Question: when did the villanova football team start


1901 Villanova Wildcats football team

Question: who is the captain of the villanova football team


1901 Villanova Wildcats football team

Question: who was the captain of the university wildcats


1992 Northwestern Wildcats football team

The 1992 Northwestern Wildcats team represented Northwestern University during the 1992 NCAA Division I-A football season . In their first year under head coach Gary Barnett , the Wildcats compiled a 3 -- 8 record ( 3 -- 5 against Big Ten Conference opponents ) and finished in ninth place in the Big Ten Conference .   The team 's offensive leaders were quarterback Len Williams with 2,110 passing yards , Dennis Lundy with 688 rushing yards , and Lee Gissendaner with 846 receiving yards . Gissendaner was also selected by the Associated Press as a first-team wide receiver on the 1992 All-Big Ten Conference football team .

Question: what conference was northwest football


/*\nCopyright (c) 2020 Pierre Marijon <pierre.marijon@hhu.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* crate use */\nuse anyhow::{anyhow, Context, Result};\nuse log::Level;\n\n/* local use */\nuse crate::error::*;\n\n#[derive(clap::Parser, Debug)]\n#[clap(\n    version = '0.1',\n    author = 'Pierre Marijon <pierre.marijon@hhu.de>',\n    about = 'KMRF: Kmer based Read Filter'\n)]\npub struct Command {\n    /// solidity bitfield produce by pcon\n    #[clap(short = 's', long = 'solidity')]\n    pub solidity: Option<String>,\n\n    /// fasta file to be correct\n    #[clap(short = 'i', long = 'inputs')]\n    pub inputs: Vec<String>,\n\n    /// path where corrected read was write\n    #[clap(short = 'o', long = 'outputs')]\n    pub outputs: Vec<String>,\n\n    /// if you want choose the minimum abundance you can set this parameter\n    #[clap(short = 'a', long = 'abundance')]\n    pub abundance: Option<u8>,\n\n    /// if a ratio of correct kmer on all kmer is lower than this threshold read is filter out, default 0.8\n    #[clap(short = 'r', long = 'ratio')]\n    pub ratio: Option<f64>,\n\n    /// if a read have length lower than this threshold read is filter out, default 1000\n    #[clap(short = 'l', long = 'min-length')]\n    pub length: Option<usize>,\n\n    /// kmer length if you didn't provide solidity path you must give a kmer length\n    #[clap(short = 'k', long = 'kmer')]\n    pub kmer: Option<u8>,\n\n    /// Number of thread use by br, 0 use all avaible core, default value 0\n    #[clap(short = 't', long = 'threads')]\n    pub threads: Option<usize>,\n\n    /// Number of sequence record load in buffer, default 8192\n    #[clap(short = 'b', long = 'record_buffer')]\n    pub record_buffer: Option<usize>,\n\n    /// verbosity level also control by environment variable BR_LOG if flag is set BR_LOG value is ignored\n    #[clap(short = 'v', long = 'verbosity', parse(from_occurrences))]\n    pub verbosity: i8,\n}\n\npub fn i82level(level: i8) -> Option<Level> {\n    match level {\n        std::i8::MIN..=0 => None,\n        1 => Some(log::Level::Error),\n        2 => Some(log::Level::Warn),\n        3 => Some(log::Level::Info),\n        4 => Some(log::Level::Debug),\n        5..=std::i8::MAX => Some(log::Level::Trace),\n    }\n}\n\npub fn read_or_compute_solidity(\n    solidity_path: Option<String>,\n    kmer: Option<u8>,\n    inputs: &[String],\n    record_buffer_len: usize,\n    abundance: Option<u8>,\n) -> Result<pcon::solid::Solid> {\n    if let Some(solidity_path) = solidity_path {\n        let solidity_reader = std::io::BufReader::new(\n            std::fs::File::open(&solidity_path)\n                .with_context(|| Error::CantOpenFile)\n                .with_context(|| anyhow!('File {:?}', solidity_path.clone()))?,\n        );\n\n        log::info!('Load solidity file');\n        Ok(pcon::solid::Solid::deserialize(solidity_reader)?)\n    } else if let Some(kmer) = kmer {\n        let mut counter = pcon::counter::Counter::new(kmer);\n\n        log::info!('Start count kmer from input');\n        for input in inputs {\n            let fasta = std::io::BufReader::new(\n                std::fs::File::open(&input)\n                    .with_context(|| Error::CantOpenFile)\n                    .with_context(|| anyhow!('File {:?}', input.clone()))?,\n            );\n\n            counter.count_fasta(fasta, record_buffer_len);\n        }\n        log::info!('End count kmer from input');\n\n        log::info!('Start build spectrum from count');\n        let spectrum = pcon::spectrum::Spectrum::from_counter(&counter);\n        log::info!('End build spectrum from count');\n\n        let abun = if let Some(a) = abundance {\n            a\n        } else {\n            log::info!('Start search threshold');\n            let abundance = spectrum\n                .get_threshold(pcon::spectrum::ThresholdMethod::FirstMinimum, 0.0)\n                .ok_or(Error::CantComputeAbundance)?;\n\n            spectrum\n                .write_histogram(std::io::stdout(), Some(abundance))\n                .with_context(|| anyhow!('Error durring write of kmer histograme'))?;\n            println!('If this curve seems bad or minimum abundance choose (marked by *) not apopriate set parameter -a');\n            log::info!('End search threshold');\n            abundance as u8\n        };\n\n        Ok(pcon::solid::Solid::from_counter(&counter, abun))\n    } else {\n        Err(anyhow!(Error::NoSolidityNoKmer))\n    }\n}\n


/*\nCopyright (c) 2020 Pierre Marijon <pierre.marijon@hhu.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* crate use */\nuse thiserror::Error;\n\n/// All error produce by Pcon\n#[derive(Debug, Error)]\npub enum Error {\n    /// We can't create file. In C binding it's equal to 0\n    #[error('We can't create file')]\n    CantCreateFile,\n\n    /// We can't open file. In C binding it's equal to 1\n    #[error('We can't open file')]\n    CantOpenFile,\n\n    /// Error durring write in file. In C binding it's equal to 2\n    #[error('Error durring write')]\n    ErrorDurringWrite,\n\n    /// Error durring read file. In C binding it's equal to 3\n    #[error('Error durring read')]\n    ErrorDurringRead,\n\n    #[error('You must provide a solidity path '-s' or a kmer length '-k'')]\n    NoSolidityNoKmer,\n\n    #[error('Can't compute minimal abundance')]\n    CantComputeAbundance,\n\n    /// No error, this exist only for C binding it's the value of a new error pointer\n    #[error('Isn't error if you see this please contact the author with this message and a description of what you do with pcon')]\n    NoError,\n}\n


/*\nCopyright (c) 2020 Pierre Marijon <pierre.marijon@hhu.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* local mod */\npub mod cli;\npub mod error;\n\n/* crates use */\nuse anyhow::{anyhow, Context, Result};\nuse rayon::iter::ParallelBridge;\nuse rayon::prelude::*;\n\n/* local use */\nuse error::*;\n\npub fn run_filter(\n    inputs: Vec<String>,\n    outputs: Vec<String>,\n    solid: pcon::solid::Solid,\n    ratio: f64,\n    length: usize,\n    record_buffer_len: usize,\n) -> Result<()> {\n    for (input, output) in inputs.iter().zip(outputs) {\n        log::info!('Start filter {} write in {}', input, output);\n\n        let mut reader = std::fs::File::open(input)\n            .with_context(|| Error::CantOpenFile)\n            .with_context(|| anyhow!('File {}', input.clone()))\n            .map(std::io::BufReader::new)\n            .map(noodles::fasta::Reader::new)?;\n\n        let mut writer = std::fs::File::create(&output)\n            .with_context(|| Error::CantCreateFile)\n            .with_context(|| anyhow!('File {}', output.clone()))\n            .map(std::io::BufWriter::new)\n            .map(noodles::fasta::Writer::new)?;\n\n        let mut iter = reader.records();\n        let mut records = Vec::with_capacity(record_buffer_len);\n\n        let mut end = false;\n        loop {\n            for _ in 0..record_buffer_len {\n                if let Some(Ok(record)) = iter.next() {\n                    records.push(record);\n                } else {\n                    end = true;\n                    break;\n                }\n            }\n\n            log::info!('Buffer len: {}', records.len());\n\n            let keeped: Vec<_> = records\n                .drain(..)\n                .par_bridge()\n                .filter_map(|record| {\n                    let l = record.sequence().len();\n                    if l < length || l < solid.k as usize {\n                        return None;\n                    }\n\n                    let mut nb_kmer = 0;\n                    let mut nb_valid = 0;\n\n                    for cano in\n                        cocktail::tokenizer::Canonical::new(record.sequence().as_ref(), solid.k)\n                    {\n                        nb_kmer += 1;\n\n                        if solid.get_canonic(cano) {\n                            nb_valid += 1;\n                        }\n                    }\n\n                    let r = (nb_valid as f64) / (nb_kmer as f64);\n\n                    if r >= ratio {\n                        Some(record)\n                    } else {\n                        None\n                    }\n                })\n                .collect();\n\n            for record in keeped {\n                writer\n                    .write_record(&record)\n                    .with_context(|| Error::ErrorDurringWrite)\n                    .with_context(|| anyhow!('File {}', output.clone()))?\n            }\n\n            records.clear();\n\n            if end {\n                break;\n            }\n        }\n        log::info!('End filter file {} write in {}', input, output);\n    }\n\n    Ok(())\n}\n\n/// Set the number of threads use by count step\npub fn set_nb_threads(nb_threads: usize) {\n    rayon::ThreadPoolBuilder::new()\n        .num_threads(nb_threads)\n        .build_global()\n        .unwrap();\n}\n


/*\nCopyright (c) 2019 Pierre Marijon <pierre.marijon@hhu.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* crate use */\nuse anyhow::Result;\nuse clap::Parser;\n\nuse kmrf::*;\n\nfn main() -> Result<()> {\n    let params = cli::Command::parse();\n\n    if let Some(level) = cli::i82level(params.verbosity) {\n        env_logger::builder()\n            .format_timestamp(Some(env_logger::fmt::TimestampPrecision::Millis))\n            .filter_level(level.to_level_filter())\n            .init();\n    } else {\n        env_logger::Builder::from_env('KMRF_LOG')\n            .format_timestamp(Some(env_logger::fmt::TimestampPrecision::Millis))\n            .init();\n    }\n\n    let ratio = if let Some(val) = params.ratio {\n        val\n    } else {\n        0.9\n    };\n\n    let length = if let Some(val) = params.length {\n        val\n    } else {\n        1000\n    };\n\n    if let Some(threads) = params.threads {\n        log::info!('Set number of threads to {}', threads);\n\n        set_nb_threads(threads);\n    }\n\n    let record_buffer = if let Some(len) = params.record_buffer {\n        len\n    } else {\n        8192\n    };\n\n    let solid = cli::read_or_compute_solidity(\n        params.solidity,\n        params.kmer,\n        &params.inputs,\n        record_buffer,\n        params.abundance,\n    )?;\n\n    kmrf::run_filter(\n        params.inputs,\n        params.outputs,\n        solid,\n        ratio,\n        length,\n        record_buffer,\n    )?;\n\n    Ok(())\n}\n


/*\nCopyright (c) 2020 Pierre Marijon <pmarijon@mpi-inf.mpg.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* std use */\nuse std::io::BufRead;\nuse std::io::Read;\nuse std::process::{Command, Stdio};\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n\n    fn diff_unorder(truth_path: &str, result_path: &str) {\n        let truth_file = std::io::BufReader::new(\n            std::fs::File::open(truth_path).expect(&format!('Impossible to open {}', truth_path)),\n        );\n\n        let mut truth: std::collections::HashSet<String> = std::collections::HashSet::new();\n\n        for res in truth_file.lines() {\n            let line = res.unwrap();\n            truth.insert(line);\n        }\n\n        let result_file = std::io::BufReader::new(\n            std::fs::File::open(result_path).expect(&format!('Impossible to open {}', result_path)),\n        );\n\n        let mut result: std::collections::HashSet<String> = std::collections::HashSet::new();\n\n        for res in result_file.lines() {\n            let line = res.unwrap();\n            result.insert(line);\n        }\n\n        if truth != result {\n            panic!(\n                'Truth {} and result {} are different',\n                truth_path, result_path\n            );\n        }\n    }\n\n    fn diff(truth_path: &str, result_path: &str) {\n        let truth_file = std::io::BufReader::new(\n            std::fs::File::open(truth_path).expect(&format!('Impossible to open {}', truth_path)),\n        );\n\n        let mut truth: Vec<String> = Vec::new();\n\n        for res in truth_file.lines() {\n            let line = res.unwrap();\n            truth.push(line);\n        }\n\n        let result_file = std::io::BufReader::new(\n            std::fs::File::open(result_path).expect(&format!('Impossible to open {}', result_path)),\n        );\n\n        let mut result: Vec<String> = Vec::new();\n\n        for res in result_file.lines() {\n            let line = res.unwrap();\n            result.push(line);\n        }\n\n        if truth != result {\n            panic!(\n                'Truth {} and result {} are different',\n                truth_path, result_path\n            );\n        }\n    }\n\n    #[test]\n    fn detection() {\n        let mut child = Command::new('./target/debug/yacrd')\n            .args(&['-i', 'tests/reads.paf', '-o', 'tests/result.yacrd'])\n            .stderr(Stdio::piped())\n            .stdout(Stdio::piped())\n            .spawn()\n            .expect('Couldn't create yacrd subprocess');\n\n        if !child.wait().expect('Error durring yacrd run').success() {\n            let mut stdout = String::new();\n            let mut stderr = String::new();\n\n            child.stdout.unwrap().read_to_string(&mut stdout).unwrap();\n            child.stderr.unwrap().read_to_string(&mut stderr).unwrap();\n\n            println!('stdout: {}', stdout);\n            println!('stderr: {}', stderr);\n            panic!();\n        }\n\n        diff_unorder('tests/truth.yacrd', 'tests/result.yacrd');\n    }\n\n    #[test]\n    fn detection_ondisk() {\n        if cfg!(windows) {\n            ()\n        } else {\n            if std::path::Path::new('tests/ondisk').exists() {\n                std::fs::remove_dir_all(std::path::Path::new('tests/ondisk'))\n                    .expect('We can't delete temporary directory of ondisk test');\n            }\n\n            std::fs::create_dir(std::path::Path::new('tests/ondisk'))\n                .expect('We can't create temporary directory for ondisk test');\n\n            let mut child = Command::new('./target/debug/yacrd')\n                .args(&[\n                    '-i',\n                    'tests/reads.paf',\n                    '-o',\n                    'tests/result.ondisk.yacrd',\n                    '-d',\n                    'tests/ondisk',\n                ])\n                .stderr(Stdio::piped())\n                .stdout(Stdio::piped())\n                .spawn()\n                .expect('Couldn't create yacrd subprocess');\n\n            if !child.wait().expect('Error durring yacrd run').success() {\n                let mut stdout = String::new();\n                let mut stderr = String::new();\n\n                child.stdout.unwrap().read_to_string(&mut stdout).unwrap();\n                child.stderr.unwrap().read_to_string(&mut stderr).unwrap();\n\n                println!('stdout: {}', stdout);\n                println!('stderr: {}', stderr);\n                panic!();\n            }\n\n            diff_unorder('tests/truth.yacrd', 'tests/result.ondisk.yacrd');\n        }\n    }\n\n    #[test]\n    fn filter() {\n        let mut child = Command::new('./target/debug/yacrd')\n            .args(&[\n                '-i',\n                'tests/reads.paf',\n                '-o',\n                'tests/result.filter.yacrd',\n                'filter',\n                '-i',\n                'tests/reads.fastq',\n                '-o',\n                'tests/reads.filter.fastq',\n            ])\n            .stderr(Stdio::piped())\n            .stdout(Stdio::piped())\n            .spawn()\n            .expect('Couldn't create yacrd subprocess');\n\n        if !child.wait().expect('Error durring yacrd run').success() {\n            let mut stdout = String::new();\n            let mut stderr = String::new();\n\n            child.stdout.unwrap().read_to_string(&mut stdout).unwrap();\n            child.stderr.unwrap().read_to_string(&mut stderr).unwrap();\n\n            println!('stdout: {}', stdout);\n            println!('stderr: {}', stderr);\n            panic!();\n        }\n\n        diff_unorder('tests/truth.yacrd', 'tests/result.filter.yacrd');\n        diff('tests/truth.filter.fastq', 'tests/reads.filter.fastq')\n    }\n\n    #[test]\n    fn extract() {\n        let mut child = Command::new('./target/debug/yacrd')\n            .args(&[\n                '-i',\n                'tests/reads.paf',\n                '-o',\n                'tests/result.extract.yacrd',\n                'extract',\n                '-i',\n                'tests/reads.fastq',\n                '-o',\n                'tests/reads.extract.fastq',\n            ])\n            .stderr(Stdio::piped())\n            .stdout(Stdio::piped())\n            .spawn()\n            .expect('Couldn't create yacrd subprocess');\n\n        if !child.wait().expect('Error durring yacrd run').success() {\n            let mut stdout = String::new();\n            let mut stderr = String::new();\n\n            child.stdout.unwrap().read_to_string(&mut stdout).unwrap();\n            child.stderr.unwrap().read_to_string(&mut stderr).unwrap();\n\n            println!('stdout: {}', stdout);\n            println!('stderr: {}', stderr);\n            panic!();\n        }\n\n        diff_unorder('tests/truth.yacrd', 'tests/result.extract.yacrd');\n        diff('tests/truth.extract.fastq', 'tests/reads.extract.fastq')\n    }\n\n    #[test]\n    fn split() {\n        let mut child = Command::new('./target/debug/yacrd')\n            .args(&[\n                '-i',\n                'tests/reads.paf',\n                '-o',\n                'tests/result.split.yacrd',\n                'split',\n                '-i',\n                'tests/reads.fastq',\n                '-o',\n                'tests/reads.split.fastq',\n            ])\n            .stderr(Stdio::piped())\n            .stdout(Stdio::piped())\n            .spawn()\n            .expect('Couldn't create yacrd subprocess');\n\n        if !child.wait().expect('Error durring yacrd run').success() {\n            let mut stdout = String::new();\n            let mut stderr = String::new();\n\n            child.stdout.unwrap().read_to_string(&mut stdout).unwrap();\n            child.stderr.unwrap().read_to_string(&mut stderr).unwrap();\n\n            println!('stdout: {}', stdout);\n            println!('stderr: {}', stderr);\n            panic!();\n        }\n\n        diff_unorder('tests/truth.yacrd', 'tests/result.split.yacrd');\n        diff('tests/truth.split.fastq', 'tests/reads.split.fastq')\n    }\n\n    #[test]\n    fn scrubb() {\n        let mut child = Command::new('./target/debug/yacrd')\n            .args(&[\n                '-i',\n                'tests/reads.paf',\n                '-o',\n                'tests/result.scrubb.yacrd',\n                'scrubb',\n                '-i',\n                'tests/reads.fastq',\n                '-o',\n                'tests/reads.scrubb.fastq',\n            ])\n            .stderr(Stdio::piped())\n            .stdout(Stdio::piped())\n            .spawn()\n            .expect('Couldn't create yacrd subprocess');\n\n        if !child.wait().expect('Error durring yacrd run').success() {\n            let mut stdout = String::new();\n            let mut stderr = String::new();\n\n            child.stdout.unwrap().read_to_string(&mut stdout).unwrap();\n            child.stderr.unwrap().read_to_string(&mut stderr).unwrap();\n\n            println!('stdout: {}', stdout);\n            println!('stderr: {}', stderr);\n            panic!();\n        }\n\n        diff_unorder('tests/truth.yacrd', 'tests/result.scrubb.yacrd');\n        diff('tests/truth.scrubb.fastq', 'tests/reads.scrubb.fastq')\n    }\n}\n


/*\n   Copyright (c) 2019 Pierre Marijon <pmarijon@mpi-inf.mpg.de>\n\n   Permission is hereby granted, free of charge, to any person obtaining a copy\n   of this software and associated documentation files (the 'Software'), to deal\n   in the Software without restriction, including without limitation the rights\n   to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n   copies of the Software, and to permit persons to whom the Software is\n   furnished to do so, subject to the following conditions:\n\n   The above copyright notice and this permission notice shall be included in all\n   copies or substantial portions of the Software.\n\n   THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n   OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n   SOFTWARE.\n*/\n\n/* crate use */\nuse anyhow::Result;\n\n/* local use */\nuse crate::reads2ovl;\n\npub struct FullMemory {\n    reads2ovl: reads2ovl::MapReads2Ovl,\n    no_overlap: Vec<(u32, u32)>,\n    read_buffer_size: usize,\n}\n\nimpl FullMemory {\n    pub fn new(read_buffer_size: usize) -> Self {\n        FullMemory {\n            reads2ovl: rustc_hash::FxHashMap::default(),\n            no_overlap: Vec::new(),\n            read_buffer_size,\n        }\n    }\n}\n\nimpl reads2ovl::Reads2Ovl for FullMemory {\n    fn get_overlaps(&mut self, new: &mut reads2ovl::MapReads2Ovl) -> bool {\n        std::mem::swap(&mut self.reads2ovl, new);\n\n        true\n    }\n\n    fn overlap(&self, id: &str) -> Result<Vec<(u32, u32)>> {\n        if let Some((vec, _)) = self.reads2ovl.get(&id.to_string()) {\n            Ok(vec.to_vec())\n        } else {\n            Ok(self.no_overlap.to_vec())\n        }\n    }\n\n    fn length(&self, id: &str) -> usize {\n        if let Some((_, len)) = self.reads2ovl.get(&id.to_string()) {\n            *len\n        } else {\n            0\n        }\n    }\n\n    fn add_overlap(&mut self, id: String, ovl: (u32, u32)) -> Result<()> {\n        self.reads2ovl\n            .entry(id)\n            .or_insert((Vec::new(), 0))\n            .0\n            .push(ovl);\n\n        Ok(())\n    }\n\n    fn add_length(&mut self, id: String, length: usize) {\n        self.reads2ovl.entry(id).or_insert((Vec::new(), 0)).1 = length;\n    }\n\n    fn add_overlap_and_length(&mut self, id: String, ovl: (u32, u32), length: usize) -> Result<()> {\n        if let Some(value) = self.reads2ovl.get_mut(&id) {\n            value.0.push(ovl);\n        } else {\n            self.reads2ovl.insert(id, (vec![ovl], length));\n        }\n\n        Ok(())\n    }\n\n    fn get_reads(&self) -> rustc_hash::FxHashSet<String> {\n        self.reads2ovl.keys().map(|x| x.to_string()).collect()\n    }\n\n    fn read_buffer_size(&self) -> usize {\n        self.read_buffer_size\n    }\n}\n


/*\nCopyright (c) 2019 Pierre Marijon <pmarijon@mpi-inf.mpg.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* crate use */\nuse anyhow::{anyhow, bail, Context, Result};\n\n/* local mod */\npub mod fullmemory;\npub mod ondisk;\n\n/* stuff declare in submod need to be accessible from mod level */\npub use self::fullmemory::*;\npub use self::ondisk::*;\n\n/* std use */\npub use self::fullmemory::*;\n\n/* local use */\nuse crate::error;\nuse crate::io;\nuse crate::util;\n\npub type MapReads2Ovl = rustc_hash::FxHashMap<String, (Vec<(u32, u32)>, usize)>;\n\npub trait Reads2Ovl {\n    fn init(&mut self, filename: &str) -> Result<()> {\n        self.sub_init(filename)\n    }\n\n    fn sub_init(&mut self, filename: &str) -> Result<()> {\n        let (input, _) = util::read_file(filename, self.read_buffer_size())?;\n\n        match util::get_file_type(filename) {\n            Some(util::FileType::Paf) => self\n                .init_paf(input)\n                .with_context(|| anyhow!('Filename: {}', filename.to_string()))?,\n            Some(util::FileType::M4) => self\n                .init_m4(input)\n                .with_context(|| anyhow!('Filename: {}', filename.to_string()))?,\n            Some(util::FileType::Fasta) => bail!(error::Error::CantRunOperationOnFile {\n                operation: 'overlap parsing'.to_string(),\n                filetype: util::FileType::Fasta,\n                filename: filename.to_string()\n            }),\n            Some(util::FileType::Fastq) => bail!(error::Error::CantRunOperationOnFile {\n                operation: 'overlap parsing'.to_string(),\n                filetype: util::FileType::Fastq,\n                filename: filename.to_string()\n            }),\n            Some(util::FileType::Yacrd) => bail!(error::Error::CantRunOperationOnFile {\n                operation: 'overlap parsing'.to_string(),\n                filetype: util::FileType::Yacrd,\n                filename: filename.to_string()\n            }),\n            None | Some(util::FileType::YacrdOverlap) => {\n                bail!(error::Error::UnableToDetectFileFormat {\n                    filename: filename.to_string()\n                })\n            }\n        }\n\n        Ok(())\n    }\n\n    fn init_paf(&mut self, input: Box<dyn std::io::Read>) -> Result<()> {\n        let mut reader = csv::ReaderBuilder::new()\n            .delimiter(b'\t')\n            .has_headers(false)\n            .flexible(true)\n            .from_reader(input);\n\n        let mut rec = csv::StringRecord::new();\n\n        while reader.read_record(&mut rec).unwrap() {\n            let record: io::PafRecord =\n                rec.deserialize(None)\n                    .with_context(|| error::Error::ReadingErrorNoFilename {\n                        format: util::FileType::Paf,\n                    })?;\n\n            let id_a = record.read_a.to_string();\n            let id_b = record.read_b.to_string();\n\n            let len_a = record.length_a;\n            let len_b = record.length_b;\n\n            let ovl_a = (record.begin_a, record.end_a);\n            let ovl_b = (record.begin_b, record.end_b);\n\n            self.add_overlap_and_length(id_a, ovl_a, len_a)?;\n            self.add_overlap_and_length(id_b, ovl_b, len_b)?;\n        }\n\n        Ok(())\n    }\n\n    fn init_m4(&mut self, input: Box<dyn std::io::Read>) -> Result<()> {\n        let mut reader = csv::ReaderBuilder::new()\n            .delimiter(b' ')\n            .has_headers(false)\n            .flexible(true)\n            .from_reader(input);\n\n        let mut rec = csv::StringRecord::new();\n\n        while reader.read_record(&mut rec).unwrap() {\n            let record: io::M4Record =\n                rec.deserialize(None)\n                    .with_context(|| error::Error::ReadingErrorNoFilename {\n                        format: util::FileType::M4,\n                    })?;\n\n            let id_a = record.read_a.to_string();\n            let id_b = record.read_b.to_string();\n\n            let len_a = record.length_a;\n            let len_b = record.length_b;\n\n            let ovl_a = (record.begin_a, record.end_a);\n            let ovl_b = (record.begin_b, record.end_b);\n\n            self.add_overlap_and_length(id_a, ovl_a, len_a)?;\n            self.add_overlap_and_length(id_b, ovl_b, len_b)?;\n        }\n\n        Ok(())\n    }\n\n    fn get_overlaps(&mut self, new: &mut MapReads2Ovl) -> bool;\n\n    fn overlap(&self, id: &str) -> Result<Vec<(u32, u32)>>;\n    fn length(&self, id: &str) -> usize;\n\n    fn add_overlap(&mut self, id: String, ovl: (u32, u32)) -> Result<()>;\n    fn add_length(&mut self, id: String, ovl: usize);\n\n    fn add_overlap_and_length(&mut self, id: String, ovl: (u32, u32), length: usize) -> Result<()>;\n\n    fn get_reads(&self) -> rustc_hash::FxHashSet<String>;\n\n    fn read_buffer_size(&self) -> usize;\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    use std::io::Write;\n\n    extern crate tempfile;\n\n    const PAF_FILE: &'static [u8] = b'1\t12000\t20\t4500\t-\t2\t10000\t5500\t10000\t4500\t4500\t255\n1\t12000\t5500\t10000\t-\t3\t10000\t0\t4500\t4500\t4500\t255\n';\n\n    const M4_FILE: &'static [u8] = b'1 2 0.1 2 0 20 4500 12000 0 5500 10000 10000\n1 3 0.1 2 0 5500 10000 12000 0 0 4500 10000\n';\n\n    #[test]\n    fn paf() {\n        let mut paf = tempfile::Builder::new()\n            .suffix('.paf')\n            .tempfile()\n            .expect('Can't create tmpfile');\n\n        paf.as_file_mut()\n            .write_all(PAF_FILE)\n            .expect('Error durring write of paf in temp file');\n\n        let mut ovl = FullMemory::new(8192);\n\n        ovl.init(paf.into_temp_path().to_str().unwrap())\n            .expect('Error in overlap init');\n\n        assert_eq!(\n            ['1'.to_string(), '2'.to_string(), '3'.to_string(),]\n                .iter()\n                .cloned()\n                .collect::<rustc_hash::FxHashSet<String>>(),\n            ovl.get_reads()\n        );\n\n        assert_eq!(vec![(20, 4500), (5500, 10000)], ovl.overlap('1').unwrap());\n        assert_eq!(vec![(5500, 10000)], ovl.overlap('2').unwrap());\n        assert_eq!(vec![(0, 4500)], ovl.overlap('3').unwrap());\n    }\n\n    #[test]\n    fn m4() {\n        let mut m4 = tempfile::Builder::new()\n            .suffix('.m4')\n            .tempfile()\n            .expect('Can't create tmpfile');\n\n        m4.as_file_mut()\n            .write_all(M4_FILE)\n            .expect('Error durring write of m4 in temp file');\n\n        let mut ovl = FullMemory::new(8192);\n\n        ovl.init(m4.into_temp_path().to_str().unwrap())\n            .expect('Error in overlap init');\n\n        assert_eq!(\n            ['1'.to_string(), '2'.to_string(), '3'.to_string(),]\n                .iter()\n                .cloned()\n                .collect::<rustc_hash::FxHashSet<String>>(),\n            ovl.get_reads()\n        );\n\n        assert_eq!(vec![(20, 4500), (5500, 10000)], ovl.overlap('1').unwrap());\n        assert_eq!(vec![(5500, 10000)], ovl.overlap('2').unwrap());\n        assert_eq!(vec![(0, 4500)], ovl.overlap('3').unwrap());\n    }\n}\n


/*\nCopyright (c) 2019 Pierre Marijon <pmarijon@mpi-inf.mpg.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* crate use */\nuse anyhow::{anyhow, Context, Result};\nuse log::info;\n\n/* local use */\nuse crate::error;\nuse crate::reads2ovl;\n\npub struct OnDisk {\n    reads2ovl: rustc_hash::FxHashMap<String, Vec<(u32, u32)>>,\n    reads2len: rustc_hash::FxHashMap<String, usize>,\n    db: sled::Db,\n    number_of_value: u64,\n    buffer_size: u64,\n    read_buffer_size: usize,\n}\n\n#[derive(Debug, Clone, serde::Deserialize)]\nstruct OnDiskRecord {\n    _begin: u32,\n    _end: u32,\n}\n\nimpl OnDisk {\n    pub fn new(on_disk_path: String, buffer_size: u64, read_buffer_size: usize) -> Self {\n        let path = std::path::PathBuf::from(on_disk_path.clone());\n\n        if let Some(parent) = path.parent() {\n            std::fs::create_dir_all(parent)\n                .with_context(|| error::Error::PathCreation { path })\n                .unwrap();\n        }\n\n        let db = sled::Config::default()\n            .path(on_disk_path)\n            .open()\n            .with_context(|| error::Error::OnDiskOpen)\n            .unwrap();\n\n        OnDisk {\n            reads2ovl: rustc_hash::FxHashMap::default(),\n            reads2len: rustc_hash::FxHashMap::default(),\n            db,\n            number_of_value: 0,\n            buffer_size,\n            read_buffer_size,\n        }\n    }\n\n    fn clean_buffer(&mut self) -> Result<()> {\n        info!(\n            'Clear cache, number of value in cache is {}',\n            self.number_of_value\n        );\n\n        let mut batch = sled::Batch::default();\n\n        for (key, vs) in self.reads2ovl.drain() {\n            let new_val: Vec<(u32, u32)> = match self\n                .db\n                .get(key.as_bytes())\n                .with_context(|| error::Error::OnDiskReadDatabase)?\n            {\n                Some(x) => {\n                    let mut orig: Vec<(u32, u32)> = bincode::deserialize(&x)\n                        .with_context(|| error::Error::OnDiskDeserializeVec)?;\n                    orig.extend(vs);\n                    orig\n                }\n                None => vs.to_vec(),\n            };\n\n            batch.insert(\n                key.as_bytes(),\n                bincode::serialize(&new_val).with_context(|| error::Error::OnDiskSerializeVec)?,\n            );\n        }\n\n        self.db\n            .apply_batch(batch)\n            .with_context(|| error::Error::OnDiskBatchApplication)?;\n\n        self.db\n            .flush()\n            .with_context(|| error::Error::OnDiskBatchApplication)?;\n\n        self.number_of_value = 0;\n\n        Ok(())\n    }\n\n    fn _overlap(&self, id: &str) -> Result<Vec<(u32, u32)>> {\n        bincode::deserialize(\n            &self\n                .db\n                .get(id.as_bytes())\n                .with_context(|| error::Error::OnDiskReadDatabase)?\n                .unwrap(),\n        )\n        .with_context(|| error::Error::OnDiskDeserializeVec)\n    }\n}\n\nimpl reads2ovl::Reads2Ovl for OnDisk {\n    fn init(&mut self, filename: &str) -> Result<()> {\n        self.sub_init(filename)?;\n\n        self.clean_buffer()\n            .with_context(|| anyhow!('Error durring creation of tempory file'))?;\n        self.number_of_value = 0;\n\n        Ok(())\n    }\n\n    fn get_overlaps(&mut self, new: &mut reads2ovl::MapReads2Ovl) -> bool {\n        let mut tmp = rustc_hash::FxHashMap::default();\n\n        if self.reads2len.is_empty() {\n            std::mem::swap(&mut tmp, new);\n            return true;\n        }\n\n        let mut remove_reads = Vec::with_capacity(self.buffer_size as usize);\n\n        for (k, v) in self.reads2len.iter().take(self.buffer_size as usize) {\n            remove_reads.push(k.clone());\n            tmp.insert(k.clone(), (self._overlap(k).unwrap(), *v));\n        }\n\n        for k in remove_reads {\n            self.reads2len.remove(&k);\n        }\n\n        std::mem::swap(&mut tmp, new);\n        false\n    }\n\n    fn overlap(&self, id: &str) -> Result<Vec<(u32, u32)>> {\n        self._overlap(id)\n    }\n\n    fn length(&self, id: &str) -> usize {\n        *self.reads2len.get(&id.to_string()).unwrap_or(&0)\n    }\n\n    fn add_overlap(&mut self, id: String, ovl: (u32, u32)) -> Result<()> {\n        self.reads2ovl.entry(id).or_insert_with(Vec::new).push(ovl);\n\n        self.number_of_value += 1;\n\n        if self.number_of_value >= self.buffer_size {\n            self.clean_buffer()?;\n        }\n\n        Ok(())\n    }\n\n    fn add_length(&mut self, id: String, length: usize) {\n        self.reads2len.entry(id).or_insert(length);\n    }\n\n    fn add_overlap_and_length(&mut self, id: String, ovl: (u32, u32), length: usize) -> Result<()> {\n        self.add_length(id.clone(), length);\n\n        self.add_overlap(id, ovl)\n    }\n\n    fn get_reads(&self) -> rustc_hash::FxHashSet<String> {\n        self.reads2len.keys().map(|x| x.to_string()).collect()\n    }\n\n    fn read_buffer_size(&self) -> usize {\n        self.read_buffer_size\n    }\n}\n


/*\nCopyright (c) 2019 Pierre Marijon <pmarijon@mpi-inf.mpg.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/* std use */\nuse std::cmp::Reverse;\n\n/* crate use */\nuse anyhow::{Context, Result};\nuse rayon::prelude::*;\n\n/* local use */\nuse crate::error;\nuse crate::reads2ovl;\nuse crate::util;\n\npub trait BadPart {\n    fn compute_all_bad_part(&mut self);\n\n    fn get_bad_part(&mut self, id: &str) -> Result<&(Vec<(u32, u32)>, usize)>;\n\n    fn get_reads(&self) -> rustc_hash::FxHashSet<String>;\n}\n\npub struct FromOverlap {\n    ovl: Box<dyn reads2ovl::Reads2Ovl>,\n    coverage: u64,\n    buffer: reads2ovl::MapReads2Ovl,\n    empty: (Vec<(u32, u32)>, usize),\n}\n\nimpl FromOverlap {\n    pub fn new(ovl: Box<dyn reads2ovl::Reads2Ovl>, coverage: u64) -> Self {\n        let empty = (Vec::new(), 0);\n        FromOverlap {\n            ovl,\n            coverage,\n            buffer: rustc_hash::FxHashMap::default(),\n            empty,\n        }\n    }\n\n    fn compute_bad_part(mut ovls: Vec<(u32, u32)>, len: usize, coverage: usize) -> Vec<(u32, u32)> {\n        let mut gaps: Vec<(u32, u32)> = Vec::new();\n        let mut stack: std::collections::BinaryHeap<Reverse<u32>> =\n            std::collections::BinaryHeap::new();\n\n        ovls.sort_unstable();\n\n        let mut first_covered = 0;\n        let mut last_covered = 0;\n\n        for interval in ovls {\n            while let Some(head) = stack.peek() {\n                if head.0 > interval.0 {\n                    break;\n                }\n\n                if stack.len() > coverage {\n                    last_covered = head.0;\n                }\n                stack.pop();\n            }\n\n            if stack.len() <= coverage {\n                if last_covered != 0 {\n                    gaps.push((last_covered, interval.0));\n                } else {\n                    first_covered = interval.0;\n                }\n            }\n            stack.push(Reverse(interval.1));\n        }\n\n        while stack.len() > coverage {\n            last_covered = stack\n                .peek()\n                .with_context(|| error::Error::NotReachableCode {\n                    name: format!('{} {}', file!(), line!()),\n                })\n                .unwrap()\n                .0;\n            if last_covered as usize >= len {\n                break;\n            }\n            stack.pop();\n        }\n\n        if first_covered != 0 {\n            gaps.insert(0, (0, first_covered));\n        }\n\n        if last_covered as usize != len {\n            gaps.push((last_covered, len as u32));\n        }\n\n        if gaps.is_empty() {\n            return gaps;\n        }\n\n        /* clean overlapped bad region */\n        let mut clean_gaps: Vec<(u32, u32)> = Vec::new();\n        let mut begin = gaps[0].0;\n        let mut end = gaps[0].1;\n        for gaps in gaps.windows(2) {\n            let g1 = gaps[0];\n            let g2 = gaps[1];\n\n            if g1.0 == g2.0 {\n                begin = g1.0;\n                end = g1.1.max(g2.1);\n            } else {\n                clean_gaps.push((begin, end));\n                begin = g2.0;\n                end = g2.1;\n            }\n        }\n        clean_gaps.push((begin, end));\n\n        clean_gaps\n    }\n}\n\nimpl BadPart for FromOverlap {\n    fn compute_all_bad_part(&mut self) {\n        let mut new = rustc_hash::FxHashMap::default();\n\n        let coverage = self.coverage as usize;\n\n        loop {\n            let finish = self.ovl.get_overlaps(&mut new);\n\n            self.buffer.extend(\n                new.drain()\n                    .par_bridge()\n                    .map(|(k, v)| (k, (FromOverlap::compute_bad_part(v.0, v.1, coverage), v.1)))\n                    .collect::<reads2ovl::MapReads2Ovl>(),\n            );\n\n            if finish {\n                break;\n            }\n        }\n    }\n\n    fn get_bad_part(&mut self, id: &str) -> Result<&(Vec<(u32, u32)>, usize)> {\n        match self.buffer.get(id) {\n            Some(v) => Ok(v),\n            None => Ok(&self.empty),\n        }\n    }\n\n    fn get_reads(&self) -> rustc_hash::FxHashSet<String> {\n        self.buffer.keys().map(|x| x.to_string()).collect()\n    }\n}\n\npub struct FromReport {\n    buffer: reads2ovl::MapReads2Ovl,\n    empty: (Vec<(u32, u32)>, usize),\n}\n\nimpl FromReport {\n    pub fn new(input_path: &str) -> Result<Self> {\n        let input =\n            std::io::BufReader::new(std::fs::File::open(input_path).with_context(|| {\n                error::Error::CantReadFile {\n                    filename: input_path.to_string(),\n                }\n            })?);\n        let mut reader = csv::ReaderBuilder::new()\n            .delimiter(b'\t')\n            .has_headers(false)\n            .from_reader(input);\n\n        let mut buffer = rustc_hash::FxHashMap::default();\n        for (line, record) in reader.records().enumerate() {\n            let result = record.with_context(|| error::Error::Reading {\n                filename: input_path.to_string(),\n                format: util::FileType::Fasta,\n            })?;\n\n            let id = result[1].to_string();\n            let len = util::str2usize(&result[2])?;\n            let bad_part = FromReport::parse_bad_string(&result[3]).with_context(|| {\n                error::Error::CorruptYacrdReport {\n                    name: input_path.to_string(),\n                    line,\n                }\n            })?;\n\n            buffer.insert(id, (bad_part, len));\n        }\n\n        let empty = (Vec::new(), 0);\n        Ok(FromReport { buffer, empty })\n    }\n\n    fn parse_bad_string(bad_string: &str) -> Result<Vec<(u32, u32)>> {\n        let mut ret = Vec::new();\n\n        if bad_string.is_empty() {\n            return Ok(ret);\n        }\n\n        for sub in bad_string.split(';') {\n            let mut iter = sub.split(',');\n            iter.next();\n\n            ret.push((\n                util::str2u32(\n                    iter.next()\n                        .with_context(|| error::Error::CorruptYacrdReportInPosition)?,\n                )?,\n                util::str2u32(\n                    iter.next()\n                        .with_context(|| error::Error::CorruptYacrdReportInPosition)?,\n                )?,\n            ));\n        }\n\n        Ok(ret)\n    }\n}\n\nimpl BadPart for FromReport {\n    fn compute_all_bad_part(&mut self) {}\n\n    fn get_bad_part(&mut self, id: &str) -> Result<&(Vec<(u32, u32)>, usize)> {\n        match self.buffer.get(id) {\n            Some(v) => Ok(v),\n            None => Ok(&self.empty),\n        }\n    }\n\n    fn get_reads(&self) -> rustc_hash::FxHashSet<String> {\n        self.buffer.keys().map(|x| x.to_string()).collect()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    use std::io::Write;\n\n    extern crate tempfile;\n    use self::tempfile::NamedTempFile;\n\n    use reads2ovl::Reads2Ovl;\n\n    #[test]\n    fn from_report() {\n        let mut report = NamedTempFile::new().expect('Can't create tmpfile');\n\n        writeln!(\n            report.as_file_mut(),\n            'NotBad	SRR8494940.65223	2706	1131,0,1131;16,2690,2706\nNotCovered	SRR8494940.141626	30116	326,0,326;27159,2957,30116\nChimeric	SRR8494940.91655	15691	151,0,151;4056,7213,11269;58,15633,15691'\n        )\n        .expect('Error durring write of report in temp file');\n\n        let mut stack = FromReport::new(report.into_temp_path().to_str().unwrap())\n            .expect('Error when create stack object');\n\n        assert_eq!(\n            [\n                'SRR8494940.65223'.to_string(),\n                'SRR8494940.141626'.to_string(),\n                'SRR8494940.91655'.to_string()\n            ]\n            .iter()\n            .cloned()\n            .collect::<rustc_hash::FxHashSet<String>>(),\n            stack.get_reads()\n        );\n\n        assert_eq!(\n            &(vec![(0, 1131), (2690, 2706)], 2706),\n            stack.get_bad_part('SRR8494940.65223').unwrap()\n        );\n        assert_eq!(\n            &(vec![(0, 326), (2957, 30116)], 30116),\n            stack.get_bad_part('SRR8494940.141626').unwrap()\n        );\n        assert_eq!(\n            &(vec![(0, 151), (7213, 11269), (15633, 15691)], 15691),\n            stack.get_bad_part('SRR8494940.91655').unwrap()\n        );\n    }\n\n    #[test]\n    fn from_overlap() {\n        let mut ovl = reads2ovl::FullMemory::new(8192);\n\n        ovl.add_overlap('A'.to_string(), (10, 990)).unwrap();\n        ovl.add_length('A'.to_string(), 1000);\n\n        ovl.add_overlap('B'.to_string(), (10, 90)).unwrap();\n        ovl.add_length('B'.to_string(), 1000);\n\n        ovl.add_overlap('C'.to_string(), (10, 490)).unwrap();\n        ovl.add_overlap('C'.to_string(), (510, 990)).unwrap();\n        ovl.add_length('C'.to_string(), 1000);\n\n        ovl.add_overlap('D'.to_string(), (0, 990)).unwrap();\n        ovl.add_length('D'.to_string(), 1000);\n\n        ovl.add_overlap('E'.to_string(), (10, 1000)).unwrap();\n        ovl.add_length('E'.to_string(), 1000);\n\n        ovl.add_overlap('F'.to_string(), (0, 490)).unwrap();\n        ovl.add_overlap('F'.to_string(), (510, 1000)).unwrap();\n        ovl.add_length('F'.to_string(), 1000);\n\n        let mut stack = FromOverlap::new(Box::new(ovl), 0);\n\n        stack.compute_all_bad_part();\n\n        assert_eq!(\n            [\n                'A'.to_string(),\n                'B'.to_string(),\n                'C'.to_string(),\n                'D'.to_string(),\n                'E'.to_string(),\n                'F'.to_string()\n            ]\n            .iter()\n            .cloned()\n            .collect::<rustc_hash::FxHashSet<String>>(),\n            stack.get_reads()\n        );\n\n        assert_eq!(\n            &(vec![(0, 10), (990, 1000)], 1000),\n            stack.get_bad_part('A').unwrap()\n        );\n        assert_eq!(\n            &(vec![(0, 10), (90, 1000)], 1000),\n            stack.get_bad_part('B').unwrap()\n        );\n        assert_eq!(\n            &(vec![(0, 10), (490, 510), (990, 1000)], 1000),\n            stack.get_bad_part('C').unwrap()\n        );\n        assert_eq!(&(vec![(990, 1000)], 1000), stack.get_bad_part('D').unwrap());\n        assert_eq!(&(vec![(0, 10)], 1000), stack.get_bad_part('E').unwrap());\n        assert_eq!(&(vec![(490, 510)], 1000), stack.get_bad_part('F').unwrap());\n    }\n\n    #[test]\n    fn coverage_upper_than_0() {\n        let mut ovl = reads2ovl::FullMemory::new(8192);\n\n        ovl.add_length('A'.to_string(), 1000);\n\n        ovl.add_overlap('A'.to_string(), (0, 425)).unwrap();\n        ovl.add_overlap('A'.to_string(), (0, 450)).unwrap();\n        ovl.add_overlap('A'.to_string(), (0, 475)).unwrap();\n\n        ovl.add_overlap('A'.to_string(), (525, 1000)).unwrap();\n        ovl.add_overlap('A'.to_string(), (550, 1000)).unwrap();\n        ovl.add_overlap('A'.to_string(), (575, 1000)).unwrap();\n\n        let mut stack = FromOverlap::new(Box::new(ovl), 2);\n\n        stack.compute_all_bad_part();\n\n        assert_eq!(&(vec![(425, 575)], 1000), stack.get_bad_part('A').unwrap());\n    }\n\n    #[test]\n    fn failled_correctly_on_corrupt_yacrd() {\n        let mut report = NamedTempFile::new().expect('Can't create tmpfile');\n\n        writeln!(\n            report.as_file_mut(),\n            'NotBad	SRR8494940.65223	2706	1131,0,1131;16,2690,2706\nNotCovered	SRR8494940.141626	30116	326,0,326;27159,2957,30116\nChimeric	SRR8494940.91655	15691	151,0,151;4056,7213,11269;58,156'\n        )\n        .unwrap();\n\n        let stack = FromReport::new(report.into_temp_path().to_str().unwrap());\n\n        if !stack.is_err() {\n            assert!(false);\n        }\n    }\n\n    #[test]\n    fn perfect_read_in_report() {\n        let mut report = NamedTempFile::new().expect('Can't create tmpfile');\n\n        writeln!(report.as_file_mut(), 'NotBad	perfect	2706	')\n            .expect('Error durring write of report in temp file');\n\n        let mut stack = FromReport::new(report.into_temp_path().to_str().unwrap())\n            .expect('Error when create stack object');\n\n        assert_eq!(\n            ['perfect'.to_string()]\n                .iter()\n                .cloned()\n                .collect::<rustc_hash::FxHashSet<String>>(),\n            stack.get_reads()\n        );\n\n        assert_eq!(&(vec![], 2706), stack.get_bad_part('perfect').unwrap());\n    }\n}\n


/*\nCopyright (c) 2018 Pierre Marijon <pmarijon@mpi-inf.mpg.de>\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the 'Software'), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n */\n\n/// Yacrd use overlap between reads, to detect 'good' and 'bad' region,\n/// a region with coverage over the threshold is 'good' others are 'bad'.\n/// If read has a 'bad' region in middle this reads is mark as 'Chimeric'.\n/// If the ratio of 'bad' region length on total read length is larger than threshold this reads is marked as 'Not_covered'.\n\n/// Yacrd can make some other actions:\n/// - filter: for sequence or overlap file, record with reads marked as Chimeric or NotCovered isn't written in the output\n/// - extract: for sequence or overlap file, record contains reads marked as Chimeric or NotCovered is written in the output\n/// - split: for sequence file bad region in the middle of reads are removed, NotCovered read is removed\n/// - scrubb: for sequence file all bad region are removed, NotCovered read is removed\n#[derive(clap::Parser, Debug)]\n#[clap(\n    version = '1.0.0 Magby',\n    author = 'Pierre Marijon <pierre@marijon.fr>',\n    name = 'yacrd'\n)]\npub struct Command {\n    /// path to input file overlap (.paf|.m4|.mhap) or yacrd report (.yacrd), format is autodetected and compression input is allowed (gz|bzip2|lzma)\n    #[clap(short = 'i', long = 'input')]\n    pub input: String,\n\n    /// path output file\n    #[clap(short = 'o', long = 'output')]\n    pub output: String,\n\n    /// number of thread use by yacrd, 0 mean all threads available, default 1\n    #[clap(short = 't', long = 'thread')]\n    pub threads: Option<usize>,\n\n    /// if coverage reach this value region is marked as bad\n    #[clap(short = 'c', long = 'coverage', default_value = '0')]\n    pub coverage: u64,\n\n    /// if the ratio of bad region length on total length is lower than this value, read is marked as NotCovered\n    #[clap(short = 'n', long = 'not-coverage', default_value = '0.8')]\n    pub not_coverage: f64,\n\n    /// Control the size of the buffer used to read paf file\n    #[clap(long = 'read-buffer-size', default_value = '8192')]\n    pub buffer_size: usize,\n\n    /// yacrd switches to 'ondisk' mode which will reduce memory usage but increase computation time. The value passed as a parameter is used as a prefix for the temporary files created by yacrd. Be careful if the prefix contains path separators (`/` for unix or `\\` for windows) this folder will be deleted\n    #[clap(short = 'd', long = 'ondisk')]\n    pub ondisk: Option<String>,\n\n    /// with the default value yacrd in 'ondisk' mode use around 1 GBytes, you can increase to reduce runtime but increase memory usage\n    #[clap(long = 'ondisk-buffer-size', default_value = '64000000')]\n    pub ondisk_buffer_size: String,\n\n    #[clap(subcommand)]\n    pub subcmd: Option<SubCommand>,\n}\n\n#[derive(clap::Parser, Debug)]\npub enum SubCommand {\n    /// All bad region of read is removed\n    #[clap()]\n    Scrubb(Scrubb),\n\n    /// Record mark as chimeric or NotCovered is filter\n    #[clap()]\n    Filter(Filter),\n\n    /// Record mark as chimeric or NotCovered is extract\n    #[clap()]\n    Extract(Extract),\n\n    /// Record mark as chimeric or NotCovered is split\n    #[clap()]\n    Split(Split),\n}\n\n#[derive(clap::Parser, Debug)]\npub struct Scrubb {\n    /// path to sequence input (fasta|fastq), compression is autodetected (none|gzip|bzip2|lzma)\n    #[clap(short = 'i', long = 'input', required = true)]\n    pub input: String,\n\n    /// path to output file, format and compression of input is preserved\n    #[clap(short = 'o', long = 'output', required = true)]\n    pub output: String,\n}\n\n#[derive(clap::Parser, Debug)]\npub struct Filter {\n    /// path to sequence input (fasta|fastq), compression is autodetected (none|gzip|bzip2|lzma)\n    #[clap(short = 'i', long = 'input', required = true)]\n    pub input: String,\n\n    /// path to output file, format and compression of input is preserved\n    #[clap(short = 'o', long = 'output', required = true)]\n    pub output: String,\n}\n\n#[derive(clap::Parser, Debug)]\npub struct Extract {\n    /// path to sequence input (fasta|fastq), compression is autodetected (none|gzip|bzip2|lzma)\n    #[clap(short = 'i', long = 'input', required = true)]\n    pub input: String,\n\n    /// path to output file, format and compression of input is preserved\n    #[clap(short = 'o', long = 'output', required = true)]\n    pub output: String,\n}\n\n#[derive(clap::Parser, Debug)]\npub struct Split {\n    /// path to sequence input (fasta|fastq), compression is autodetected (none|gzip|bzip2|lzma)\n    #[clap(short = 'i', long = 'input', required = true)]\n    pub input: String,\n\n    /// path to output file, format and compression of input is preserved\n    #[clap(short = 'o', long = 'output', required = true)]\n    pub output: String,\n}\n


Question: what is the use of fn key in mac

It is typically found on laptops due to their keyboard size restrictions .


Question: when did the ipod 7th generation come out

Cupertino , California : Apple .


Question: who dies in season 6 once upon a time

^ Jump up to : Abrams , Natalie ( August 5 , 2016 ) .


Question: who was the first avatar ever in the last airbender

Jeremy Zuckerman and Benjamin Wynn composed the series ' music and sound ; they experimented with a wide range of instruments , including the guzheng , pipa , and duduk , to match the show 's Asia - influenced setting .


Question: how many games do they need to win to win the world series

p. 21 .


Question: when did rufus wainwright record across the universe

266 -- 67 Jump up ^ Sheff , p. 266 Jump up ^ Karimi , Faith .


Question: two core functions of the world health organization (who) are

In 1977 , the first list of essential medicines was drawn up , and a year later the ambitious goal of `` health for all '' was declared .


Question: when did adam smith published the wealth of nations

Boston Review .


Question: how many english soldiers died in world war 2

Statistics published in Russia list civilian war losses 6,074,857 civilians killed reported by the Extraordinary State Commission in 1946 641,803 famine deaths during the siege of Leningrad according to official figures 58,000 killed in bombing raids ( 40,000 Stalingrad , 17,000 Leningrad and 1,000 Moscow ) and an additional 645,000 civilian reservists that were killed or captured are also included with civilian casualties .


def run(self):
        """
        Called when the process intializes.
        """
        logger.info('started worker')

        FULL_NAMESPACE = settings.FULL_NAMESPACE
        MINI_NAMESPACE = settings.MINI_NAMESPACE
        MAX_RESOLUTION = settings.MAX_RESOLUTION
        full_uniques = FULL_NAMESPACE + 'unique_metrics'
        mini_uniques = MINI_NAMESPACE + 'unique_metrics'
        pipe = self.redis_conn.pipeline()

        while 1:

            # Make sure Redis is up
            try:
                self.redis_conn.ping()
            except:
                logger.error('worker can\'t connect to redis at socket path %s' % settings.REDIS_SOCKET_PATH)
                sleep(10)
                self.redis_conn = StrictRedis(unix_socket_path = settings.REDIS_SOCKET_PATH)
                pipe = self.redis_conn.pipeline()
                continue

            try:
                # Get a chunk from the queue with a 15 second timeout
                chunk = self.q.get(True, 15)
                now = time()

                for metric in chunk:

                    # Check if we should skip it
                    if self.in_skip_list(metric[0]):
                        continue

                    # Bad data coming in
                    if metric[1][0] < now - MAX_RESOLUTION:
                        continue

                    # Append to messagepack main namespace
                    key = ''.join((FULL_NAMESPACE, metric[0]))
                    pipe.append(key, packb(metric[1]))
                    pipe.sadd(full_uniques, key)

                    if not self.skip_mini:
                        # Append to mini namespace
                        mini_key = ''.join((MINI_NAMESPACE, metric[0]))
                        pipe.append(mini_key, packb(metric[1]))
                        pipe.sadd(mini_uniques, mini_key)

                    pipe.execute()

                # Log progress
                if self.canary:
                    logger.info('queue size at %d' % self.q.qsize())
                    self.send_graphite_metric('skyline.horizon.queue_size', self.q.qsize())

            except Empty:
                logger.info('worker queue is empty and timed out')
            except WatchError:
                logger.error(key)
            except __HOLE__:
                pass
            except Exception as e:
                logger.error("worker error: " + str(e))


def new_id(self, i, l):
        try:
            return l[i]
        except __HOLE__:
            l[i] = len(l)
            return l[i]


@property
    def storage_path(self):
        try:
            conan_user_home = os.getenv("CONAN_USER_HOME")
            if conan_user_home:
                storage = self.storage["path"]
                if storage[:2] == "~/":
                    storage = storage[2:]
                result = os.path.join(conan_user_home, storage)
            else:
                result = os.path.expanduser(self.storage["path"])
        except __HOLE__:
            result = None
        result = get_env('CONAN_STORAGE_PATH', result)
        return result


def run_tests(tests, test_input, timeout = 60, waittime = 0.1):
    """
    Runs all the tests given by directly executing them. If they return
    with a non-zero exit code, it adds them to the dictionary returned.

    Keyword arguments:
        tests -- a list of paths to executable test files
        test_input -- dictionary of key value pairs to provide to the test
        timeout -- how many seconds to let the tests run
        waittime -- how many seconds to wait in between test polls
    """
    failed_tests = {}
    for t in tests:
        log.debug("Executing test %s", t)

        p = subprocess.Popen(t, stdin=subprocess.PIPE)

        to_send = json.dumps(test_input)
        log.debug("Sending test_input to test: %s", to_send)

        # Using write here instead of communicate because communicate
        # blocks on reading output
        try:
            p.stdin.write(to_send + os.linesep)
            p.stdin.close()

            countdown = int(timeout / waittime)
            while countdown:
                retcode = p.poll()

                if retcode is not None:
                    break

                time.sleep(waittime)
                countdown = countdown - 1
            else:
                p.terminate()
                log.warn("Test %s timed out", t)
                p.wait()
                retcode = 'TIMEOUT'
        except __HOLE__ as e:
            log.warn("Caught IOError from process: %s", e)
            retcode = e

        if retcode:
            failed_tests[t] = retcode
            log.error("Test %s failed with code %s", t, retcode)

    return failed_tests


def git_version():
    def _minimal_ext_cmd(cmd):
        # construct minimal environment
        env = {}
        for k in ['SYSTEMROOT', 'PATH']:
            v = os.environ.get(k)
            if v is not None:
                env[k] = v
        # LANGUAGE is used on win32
        env['LANGUAGE'] = 'C'
        env['LANG'] = 'C'
        env['LC_ALL'] = 'C'
        out = subprocess.Popen(
            cmd, stdout=subprocess.PIPE, env=env).communicate()[0]
        return out

    try:
        out = _minimal_ext_cmd(['git', 'rev-parse', 'HEAD'])
        GIT_REVISION = out.strip().decode('ascii')
    except __HOLE__:
        GIT_REVISION = "Unknown"

    return GIT_REVISION

# This is a bit hackish: we are setting a global variable so that the main
# pyfunt __init__ can detect if it is being loaded by the setup routine, to
# avoid attempting to load components that aren't built yet.  While ugly, it's
# a lot more robust than what was previously being used.


def setup_package():

    # Rewrite the version file every time

    write_version_py()
    cmdclass = {}

    # Figure out whether to add ``*_requires = ['numpy']``.
    # We don't want to do that unconditionally, because we risk updating
    # an installed numpy which fails too often.  Just if it's not installed, we
    # may give it a try.  See gh-3379.
    build_requires = []
    try:
        import numpy
        if (len(sys.argv) >= 2 and sys.argv[1] == 'bdist_wheel' and
                sys.platform == 'darwin'):
            # We're ony building wheels for platforms where we know there's
            # also a Numpy wheel, so do this unconditionally.  See gh-5184.
            build_requires = ['numpy>=1.7.1']
    except:
        build_requires = ['numpy>=1.7.1']

    metadata = dict(
        name="pyfunt",
        author="Daniele Ettore Ciriello",
        author_email="ciriello.daniele@gmail.com",
        version="0.1",
        license="MIT",
        url="https://github.com/dnlcrl/PyFunt",
        download_url="https://github.com/dnlcrl/PyFunt",
        description="Pythonic Deep Learning Framework",
        packages=['pyfunt', 'pyfunt/layers', 'pyfunt/utils'],
        cmdclass=cmdclass,  # {'build_ext': build_ext},
        platforms=["Windows", "Linux", "Solaris", "Mac OS-X", "Unix"],
        setup_requires=build_requires,
        install_requires=required,
        # ext_modules=extensions,
        keywords='pyfunt deep learning artificial neural network convolution',
    )

    if len(sys.argv) >= 2 and ('--help' in sys.argv[1:] or
                               sys.argv[1] in ('--help-commands', 'egg_info', '--version',
                                               'clean')):
        # For these actions, NumPy is not required.
        #
        # They are required to succeed without Numpy for example when
        # pip is used to install Scipy when Numpy is not yet present in
        # the system.
        try:
            from setuptools import setup
        except __HOLE__:
            from distutils.core import setup
    else:
        if (len(sys.argv) >= 2 and sys.argv[1] in ('bdist_wheel', 'bdist_egg')) or (
                'develop' in sys.argv):
            # bdist_wheel/bdist_egg needs setuptools
            import setuptools

        from numpy.distutils.core import setup

        cwd = os.path.abspath(os.path.dirname(__file__))
        if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
            # Generate Cython sources, unless building from source release
            generate_cython()

        metadata['configuration'] = configuration

    print 'setup complete'
    setup(**metadata)


def exclude(self, excludes):
        """Takes a list of node names and removes all nodes and their
        successors from the graph.

        :param excludes: list of node names
        :type excludes: list of string
        """
        if not excludes:
            return
        if not isinstance(excludes, (list, tuple)):
            excludes = [excludes]

        excludes = set(excludes)
        # index the nodes by name
        names2nodes = {}
        for node in self.nodes():
            if node._job.name is not None:
                names2nodes[node._job.name] = node

        def _recursive_remove(node, force=True):
            parents = list(node.parents())
            if force or len(parents) <= 1:
                children = list(node.children())
                map(lambda n: _recursive_remove(n, False),
                    children)
                try:
                    log.info("Excluding node %s", node)
                    self.remove(node)
                    # check the children again, they might have becom invalid
                    for child in [c for c in children
                                  if c._tool in self._nodes]:
                        try:
                            child._tool.validate()
                        except:
                            log.info("Forcing exclude of %s, "
                                     "node became invalid",
                                     child)
                            _recursive_remove(child)
                except __HOLE__:
                    ## ignore errors where the node was already removed
                    pass

        for name in excludes:
            if not name in names2nodes:
                log.warn("Node marked for exclusing not found: %s", name)
            else:
                if isinstance(name, basestring) and not name in names2nodes:
                    node = names2nodes[name]
                else:
                    node = name
                _recursive_remove(names2nodes[name])
        self._update_cleanup_nodes()


def download(uri, filename=None, headers=None, redirect_limit=5, **kwargs):
    """ Get a remote resource and save the content to a local file. If a local
    file already exists at the download destination, the modification time of
    that file is sent in the request headers as the `If-Modified-Since` value
    to perform a conditional GET. If the remote file has not been modified, a
    304 response should then be returned.

    :param uri:
    :param filename:
    :param headers:
    :param redirect_limit:
    :param kwargs:
    :return:
    """
    from datetime import datetime
    from os import utime
    from os.path import getmtime
    from time import time
    from .tardis import datetime_to_timestamp
    try:
        last_modified = datetime.fromtimestamp(getmtime(filename))
    except __HOLE__:
        last_modified = None
    with get(uri, if_modified_since=last_modified, headers=headers, redirect_limit=redirect_limit, **kwargs) as source:
        if source.status_code == 200:
            if not filename:
                filename = source.filename
            with open(filename, "wb") as destination:
                finished = False
                while not finished:
                    data = source.read(8192)
                    if data:
                        destination.write(data)
                    else:
                        finished = True
            utime(filename, (time(), datetime_to_timestamp(source.last_modified)))
            return True
        elif source.status_code == 304:
            return False
        else:
            raise ValueError("Unexpected status code %s from download response" % source.status_code)


def ReadFile(filename, logger=None):
  """Read the contents of the file.

  An optional logger can be specified to emit messages to your favorite logging
  stream. If specified, then no exception is raised. This is external so that it
  can be used by third-party applications.

  Arguments:
    filename: (unicode) The name of the file.
    logger: (function) A function or lambda that takes a string and emits it.

  Returns:
    The contents of filename.

  Raises:
    IOError: raised if there was an error reading the file.
  """
  try:
    with open(filename, 'rb') as fd:
      encoding = tokenize.detect_encoding(fd.readline)[0]
  except __HOLE__ as err:
    if logger:
      logger(err)
    raise

  try:
    with py3compat.open_with_encoding(filename,
                                      mode='r',
                                      encoding=encoding) as fd:
      source = fd.read()
    return source, encoding
  except IOError as err:
    if logger:
      logger(err)
    raise


The condensed consolidated statements of operations for the three and six months ended May 31 , 2016 are not necessarily indicative of the results to be expected for the full year .


As a result of this change in management structure , the Company re - evaluated its reportable segments and determined that neither operating segment met the reportable criteria set forth in Accounting Standards Codification ( " ASC " ) 280 , Segment Reporting .


All prior year segment information has been restated to conform with the 2016 presentation .


Language models have become a key step to achieve state-of-the-art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized word representations such as OpenAI GPT (Radford et al., 2018), BERT (Devlin et al., 2019), or XLNet (Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to complex NLP tasks (natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks are shared with the research community for further reproducible experiments in French NLP.

Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized word representations such as OpenAI GPT (Radford et al., 2018), BERT (Devlin et al., 2019), or XLNet (Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to complex NLP tasks (natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks , called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.


Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized word representations such as OpenAI GPT (Radford et al., 2018 ), BERT ( Devlin et al., 2019 ), or XLNet ( Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to complex NLP tasks (  natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.

Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018 ; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019 ; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks ( text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.


An effective method to generate a large number of parallel sentences for training improved neural machine translation (NMT) systems is the use of  back-translations of the target-side monolingual data. The method was not able to utilize the available huge amount of  monolingual data because of the inability of  models to differentiate between the authentic and synthetic parallel data  . Tagging, or using gates, has been used to enable translation models to distinguish between synthetic and authentic data, improving standard back-translation and also enabling the use of iterative back-translation on language pairs that under-performed using standard back-translation. This workpresents  pre-training and fine-tuning as a simplified but more effective approach of differentiating between the two data. The approach - tag-less back-translation - trains the model on the synthetic data and fine-tunes it on the authentic data. Experiments have shown the approach to outperform the baseline and standard back-translation by 4.0 and 0.7 BLEU respectively on low resource English-Vietnamese NMT. While the need for tagging (noising) the dataset has been removed, the technique outperformed tagged back-translation by 0.4 BLEU. The approach reached the best scores in less training time than the standard and tagged back-translation approaches .

An effective method to generate a large number of parallel sentences for training improved neural machine translation (NMT) systems is the use of the back-translations of the target-side monolingual data. The standard back-translation method has been shown to be unable to efficiently utilize the available huge amount of existing monolingual data because of the inability of translation models to differentiate between the authentic and synthetic parallel data during training . Tagging, or using gates, has been used to enable translation models to distinguish between synthetic and authentic data, improving standard back-translation and also enabling the use of iterative back-translation on language pairs that underperformed using standard back-translation. In this work, we approach back-translation as a domain adaptation problem, eliminating the need for explicit tagging. In the approach --tag-less back-translation -- the synthetic and authentic parallel data are treated as out-of-domain and in-domain data respectively and, through pre-training and fine-tuning, the translation model is shown to be able to learn more efficiently from them during training. Experimental results have shown that the approach outperforms the standard and tagged back-translation approaches on low resource English-Vietnamese and English-German neural machine translation .


Authorship attribution is the process of identifying the author of a text. Classification-based approaches work well for small numbers of candidate authors, but only similarity-based methods are applicable for larger numbers of authors or for authors beyond the training set. While deep learning methods have been applied to classification-based approaches, current similarity-based  methods only embody static notions of similarity. Siamese networks have been used to develop learned notions of similarity in one-shot image tasks, and also for tasks of  semantic relatedness in NLP. We examine their application to the stylistic task of authorship attribution , and show that they can substantially outperform both classification- and existing similarity-based approaches on datasets with large numbers of authors .

Authorship attribution is the process of identifying the author of a text. Classification-based approaches work well for small numbers of candidate authors, but only similarity-based methods are applicable for larger numbers of authors or for authors beyond the training set. While deep learning methods have been applied to classification-based approaches, applications to similarity-based applications have been limited, and most similarity-based methods only embody static notions of similarity. Siamese networks have been used to develop learned notions of similarity in one-shot image tasks, and also for tasks of mostly semantic relatedness in NLP. We examine their application to the stylistic task of authorship attribution on datasets with large numbers of authors, looking at multiple energy functions and neural network architectures, and show that they can substantially outperform both classification- and existing similarity-based approaches . We also find an unexpected relationship between choice of energy function and number of authors, in terms of performance .