How to return an Iterator generated by a function that accepts & 'mut self (when self is created locally)?

Question

Update: The message header has been updated and the answer has been removed from the question. Short answer: you cannot. Please see my answer to this question.

I am following the blog post about the error here (github for here ), and I tried to make some changes to the code, so the search function returns an Iterator instead of Vec . It was insanely complicated and I was stuck.

I got to this point:

 fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &str) -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, FnMut(Result<Row, csv::Error>) -> Option<Result<PopulationCount, csv::Error>>>, CliError> { let mut found = vec![]; let input: Box<io::Read> = match *file_path { None => Box::new(io::stdin()), Some(ref file_path) => Box::new(try!(fs::File::open(file_path))), }; let mut rdr = csv::Reader::from_reader(input); let closure = |row: Result<Row, csv::Error>| -> Option<Result<PopulationCount, csv::Error>> { let row = match row { Ok(row) => row, Err(err) => return Some(Err(From::from(err))), }; match row.population { None => None, Some(count) => if row.city == city { Some(Ok(PopulationCount { city: row.city, country: row.country, count: count, })) } else { None } } }; let found = rdr.decode::<Row>().filter_map(closure); if !found.all(|row| match row { Ok(_) => true, _ => false, }) { Err(CliError::NotFound) } else { Ok(found) } } 

with the following compiler error:

 src/main.rs:97:1: 133:2 error: the trait `core::marker::Sized` is not implemented for the type `core::ops::FnMut(core::result::Result<Row, csv::Error>) -> core::option::Option<core::result::Result<PopulationCount, csv::Error>>` [E0277] src/main.rs:97 fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &str) -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, FnMut(Result<Row, csv::Error>) -> Option<Result<PopulationCount, csv::Error>>>, CliError> { src/main.rs:98 let mut found = vec![]; src/main.rs:99 let input: Box<io::Read> = match *file_path { src/main.rs:100 None => Box::new(io::stdin()), src/main.rs:101 Some(ref file_path) => Box::new(try!(fs::File::open(file_path))), src/main.rs:102 }; ... src/main.rs:97:1: 133:2 note: `core::ops::FnMut(core::result::Result<Row, csv::Error>) -> core::option::Option<core::result::Result<PopulationCount, csv::Error>>` does not have a constant size known at compile-time src/main.rs:97 fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &str) -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, FnMut(Result<Row, csv::Error>) -> Option<Result<PopulationCount, csv::Error>>>, CliError> { src/main.rs:98 let mut found = vec![]; src/main.rs:99 let input: Box<io::Read> = match *file_path { src/main.rs:100 None => Box::new(io::stdin()), src/main.rs:101 Some(ref file_path) => Box::new(try!(fs::File::open(file_path))), src/main.rs:102 }; ... error: aborting due to previous error 

I also tried defining this function:

 fn search<'a, P: AsRef<Path>, F>(file_path: &Option<P>, city: &str) -> Result<FilterMap<csv::reader::DecodedRecords<'a, Box<Read>, Row>, F>, CliError> where F: FnMut(Result<Row, csv::Error>) -> Option<Result<PopulationCount, csv::Error>> { 

with these compiler errors:

 src/main.rs:131:12: 131:17 error: mismatched types: expected `core::iter::FilterMap<csv::reader::DecodedRecords<'_, Box<std::io::Read>, Row>, F>`, found `core::iter::FilterMap<csv::reader::DecodedRecords<'_, Box<std::io::Read>, Row>, [closure src/main.rs:105:19: 122:6]>` (expected type parameter, found closure) [E0308] src/main.rs:131 Ok(found) 

I cannot close Box , because then filter_map will not be accepted.

Then I tried this:

 fn search<'a, P: AsRef<Path>>(file_path: &Option<P>, city: &'a str) -> Result<(Box<Iterator<Item=Result<PopulationCount, csv::Error>> + 'a>, csv::Reader<Box<io::Read>>), CliError> { let input: Box<io::Read> = match *file_path { None => box io::stdin(), Some(ref file_path) => box try!(fs::File::open(file_path)), }; let mut rdr = csv::Reader::from_reader(input); let mut found = rdr.decode::<Row>().filter_map(move |row| { let row = match row { Ok(row) => row, Err(err) => return Some(Err(err)), }; match row.population { None => None, Some(count) if row.city == city => { Some(Ok(PopulationCount { city: row.city, country: row.country, count: count, })) }, _ => None, } }); if found.size_hint().0 == 0 { Err(CliError::NotFound) } else { Ok((box found, rdr)) } } fn main() { let args: Args = Docopt::new(USAGE) .and_then(|d| d.decode()) .unwrap_or_else(|err| err.exit()); match search(&args.arg_data_path, &args.arg_city) { Err(CliError::NotFound) if args.flag_quiet => process::exit(1), Err(err) => fatal!("{}", err), Ok((pops, rdr)) => for pop in pops { match pop { Err(err) => panic!(err), Ok(pop) => println!("{}, {}: {} - {:?}", pop.city, pop.country, pop.count, rdr.byte_offset()), } } } } 

Which gives me this error:

 src/main.rs:107:21: 107:24 error: `rdr` does not live long enough src/main.rs:107 let mut found = rdr.decode::<Row>().filter_map(move |row| { ^~~ src/main.rs:100:117: 130:2 note: reference must be valid for the lifetime 'a as defined on the block at 100:116... src/main.rs:100 -> Result<(Box<Iterator<Item=Result<PopulationCount, csv::Error>> + 'a>, csv::Reader<Box<io::Read>>), CliError> { src/main.rs:101 let input: Box<io::Read> = match *file_path { src/main.rs:102 None => box io::stdin(), src/main.rs:103 Some(ref file_path) => box try!(fs::File::open(file_path)), src/main.rs:104 }; src/main.rs:105 ... src/main.rs:106:51: 130:2 note: ...but borrowed value is only valid for the block suffix following statement 1 at 106:50 src/main.rs:106 let mut rdr = csv::Reader::from_reader(input); src/main.rs:107 let mut found = rdr.decode::<Row>().filter_map(move |row| { src/main.rs:108 let row = match row { src/main.rs:109 Ok(row) => row, src/main.rs:110 Err(err) => return Some(Err(err)), src/main.rs:111 }; ... error: aborting due to previous error 

Am I doing something wrong, or am I mistaken? Did I miss something really simple and stupid? I'm not sure where to go from here.

Answer 1

This answer is based on @bluss answer + help from #rust on irc.mozilla.org

One of the problems, not obvious from the code and causing the final error displayed a little higher, is related to the definition of csv::Reader::decode (see source ). It takes &'a mut self , an explanation of this problem is discussed in this. This essentially makes the reader's lifetime be limited to the block into which he entered. A way to fix this is to split the function in half (since I cannot control the definition of the function, as recommended in the previous answer link). It took me a lifetime for the reader to be valid in the main function, so the reader could then be passed to the search function. See the code below (it could have been cleared more):

 fn population_count<'a, I>(iter: I, city: &'a str) -> Box<Iterator<Item=Result<PopulationCount,csv::Error>> + 'a> where I: IntoIterator<Item=Result<Row,csv::Error>>, I::IntoIter: 'a, { Box::new(iter.into_iter().filter_map(move |row| { let row = match row { Ok(row) => row, Err(err) => return Some(Err(err)), }; match row.population { None => None, Some(count) if row.city == city => { Some(Ok(PopulationCount { city: row.city, country: row.country, count: count, })) }, _ => None, } })) } fn get_reader<P: AsRef<Path>>(file_path: &Option<P>) -> Result<csv::Reader<Box<io::Read>>, CliError> { let input: Box<io::Read> = match *file_path { None => Box::new(io::stdin()), Some(ref file_path) => Box::new(try!(fs::File::open(file_path))), }; Ok(csv::Reader::from_reader(input)) } fn search<'a>(reader: &'a mut csv::Reader<Box<io::Read>>, city: &'a str) -> Box<Iterator<Item=Result<PopulationCount, csv::Error>> + 'a> { population_count(reader.decode::<Row>(), city) } fn main() { let args: Args = Docopt::new(USAGE) .and_then(|d| d.decode()) .unwrap_or_else(|err| err.exit()); let reader = get_reader(&args.arg_data_path); let mut reader = match reader { Err(err) => fatal!("{}", err), Ok(reader) => reader, }; let populations = search(&mut reader, &args.arg_city); let mut found = false; for pop in populations { found = true; match pop { Err(err) => fatal!("fatal !! {}", err), Ok(pop) => println!("{}, {}: {}", pop.city, pop.country, pop.count), } } if !(found || args.flag_quiet) { fatal!("{}", CliError::NotFound); } } 

I learned a lot by trying to get this to work, and got a big appreciation for compiler errors. Now it’s clear that if it were C, the last error above could actually cause segfaults, which would be much harder to debug. I also realized that converting from a pre-computed vec to an iterator requires more active thinking about when memory comes in and out of scope; I can't just change a few function calls and return types and call them day.

Answer 2

Returning iterators are possible, but with some limitations.

To demonstrate this, two examples: (A) with an explicit iterator type and (B) using a box (playpen link).

 use std::iter::FilterMap; fn is_even(elt: i32) -> Option<i32> { if elt % 2 == 0 { Some(elt) } else { None } } /// (A) pub fn evens<I: IntoIterator<Item=i32>>(iter: I) -> FilterMap<I::IntoIter, fn(I::Item) -> Option<I::Item>> { iter.into_iter().filter_map(is_even) } /// (B) pub fn cumulative_sums<'a, I>(iter: I) -> Box<Iterator<Item=i32> + 'a> where I: IntoIterator<Item=i32>, I::IntoIter: 'a, { Box::new(iter.into_iter().scan(0, |acc, x| { *acc += x; Some(*acc) })) } fn main() { // The output is: // 0 is even, 10 is even, // 1, 3, 6, 10, for even in evens(vec![0, 3, 7, 10]) { print!("{} is even, ", even); } println!(""); for cs in cumulative_sums(1..5) { print!("{}, ", cs); } println!(""); } 

You are having a problem with (A) - explicit type! Unboxed closures that we get from regular lambda expressions with the syntax |a, b, c| .. |a, b, c| .. , have unique anonymous types. Functions require explicit return types, so it doesn't work here.

Some solutions for returning faults:

• Use the function pointer fn() , as in example (A). Often you do not need a closing environment.
• Close the cover. This is reasonable, even if iterators do not currently support it. Not your mistake.
• Insert iterator
• Returns a custom iterator structure. Some template is required.

You can see that in example (B) we have to be careful with the lifetimes. He says the return value is Box<Iterator<Item=i32> + 'a> , what is that 'a ? This is the shortest life expectancy required of everything inside the box! We also put 'a on I::IntoIter - this ensures that we can put this inside the field.

If you just say Box<Iterator<Item=i32>> , it will accept 'static .

We must explicitly declare the lifetimes of the contents of our window. Just to be safe.

This is actually the main problem with your function. You have the following: DecodedRecords<'a, Box<Read>, Row>, F>

Look, 'a ! This type is borrowing something. The problem is that he does not borrow it from the source data. There is no 'a at the inputs.

You will understand that it takes from the value that you create during the function, and this lifetime expires when the function returns. We cannot return DecodedRecords<'a> from a function because it wants to borrow a local variable .

Where to go from here? The simplest answer is to perform the same split as csv. One part (Struct or value) to which the reader belongs, and one part (structure or value), which is an iterator, is borrowed from the reader.

Perhaps the csv box has its own decoder, which takes on the role of the reader that it processes. In this case, you can use this to fix the borrowing problem.