Update

New ideas that came out of conversation yesterday:

-Package up the point cloud datasets - a few examples of each, with kidnapping events identified - and send it out for second opinion.

-Try plotting the mean/stdev datasets from before; if you can't identify a trend or cutoff point, the NN won't be able to, either.

-Figure out what the covariance matrix represents, and why the first element has been so useful. Might give insight on how to study it.

Same Topic, Various Guises

I'm reading "Writing Your Dissertation in 15 Minutes A Day" by Joan Bolker, and I liked one of the parts enough to blog about it:

"Some people seem always to have known what they want to write their dissertations about. They are the lucky ones...Some, like me, have written their way through the same topic in various guises often enough so they know it's theirs for life."

The second sentence there stuck out to me, because whether it's robots or people, pulling information from messy data sets appears to be kind of my schtick when it comes to research.  For one of my final projects in college (I had 2 - one for Applied Math, one for CompSci), I obtained several decades' worth of U.S. government census data for each of the Saint Louis Metropolitan Area's counties and created a model for the population in-flows and out-flows from each county (spoiler: STL City and County had serious out-flows to St Charles and Jefferson County).  I was really proud of the big Excel spreadsheets I made - for the time span I was looking at, Census data was spread out among a couple websites.  Each source had its own file format, so it took a lot of data cleaning to get a pretty dataset with all the information I needed.  It felt easier to write the programs that manipulated the datasets and distilled their information into a single end result, but I remember that getting the model right was a challenge, too.

There are parallels to this in my current thesis project for finishing my M.S. - again, it took forever to put together the infrastructure that collects robot data.  The thing that makes this more complicated is instead of working with a finite dataset, I'm constantly finding that I need A) more examples of the driving route, and B) different routes to compare against. 

Unfortunately, I haven't found a good way to automate the ROS/Python/bash scripts so that all the ROS code I need runs in parallel, and that means there's a lot of set-up each time I want to collect more driving data.  The good part is that I have some solid bash/Python scripts that automate the data cleaning and formatting part after the data has been collected.

Conclusion: It might be worth spending a little more time on getting the data collection infrastructure automated, if that would speed up the rest of the process.

Coding For Slope

This is a covariance[0] plot for a no-kidnapping data collection trial:

I first implemented what I talked about in the last post: I took 3 windows of 3 prior points a piece and averaged each window into a number.  If the previous window's average was bigger, I coded this as a 0. Otherwise, it was a 1.

This didn't work so well - there were more 1's than I wanted to see in a normal, no-kidnapping dataset. 

The next thing I tried was taking the same 9 previous points and comparing them consecutively - if prev_point9 > prev_point8, it's a 0, and otherwise a 1.  That resulted in the dataset below. 


I like this one because it represents short-lived spikes and longer-term increases.

Weaknesses of this modeling approach:
1. Doesn't represent the intensity of the increase (magnitude of the slope). I think it's mostly the degree of the slope that differentiates a kidnapping instance's covariance spike from a regular localization covariance spike. 
2. I'd like a way of identifying "This timestep and the 5 previous timesteps were ALL 1's" - somehow, that needs to make it into the model.

Time/Covariance[0]	t9>8	t 8>7	t7>6	t 6>5	t 5>4	t 4>3	t 3>2	t 2>1
32.68	0	0	0	0	0	0	0	0
43	0	0	0	0	0	0	0	1
43.36	0	0	0	0	0	0	1	0
43.64	0	0	0	0	0	1	0	0
44.18	0	0	0	0	1	0	0	0
50.3	0	0	0	1	0	0	0	0
50.57	0	0	1	0	0	0	0	0
54	0	1	0	0	0	0	0	0
56.17	1	0	0	0	0	0	0	0
57.52	0	0	0	0	0	0	0	0
58.44	0	0	0	0	0	0	0	0
61.6	0	0	0	0	0	0	0	0
62.4	0	0	0	0	0	0	0	0
65.14	0	0	0	0	0	0	0	0
69.98	0	0	0	0	0	0	0	1
73.63	0	0	0	0	0	0	1	0
73.93	0	0	0	0	0	1	0	0
74.33	0	0	0	0	1	0	0	0
77.55	0	0	0	1	0	0	0	0
80.56	0	0	1	0	0	0	0	1
81.55	0	1	0	0	0	0	1	1
85.61	1	0	0	0	0	1	1	1
86.2	0	0	0	0	1	1	1	1
88.1	0	0	0	1	1	1	1	1
89.54	0	0	1	1	1	1	1	0
91.81	0	1	1	1	1	1	0	0
93.91	1	1	1	1	1	0	0	1
94.47	1	1	1	1	0	0	1	1
95.91	1	1	1	0	0	1	1	0
97.32	1	1	0	0	1	1	0	0
102.73	1	0	0	1	1	0	0	1
104.1	0	0	1	1	0	0	1	1
104.61	0	1	1	0	0	1	1	1
105.72	1	1	0	0	1	1	1	1
107.29	1	0	0	1	1	1	1	0
108.7	0	0	1	1	1	1	0	1
110.17	0	1	1	1	1	0	1	1
111.98	1	1	1	1	0	1	1	1
113.45	1	1	1	0	1	1	1	0
114.23	1	1	0	1	1	1	0	0
114.82	1	0	1	1	1	0	0	0
118.75	0	1	1	1	0	0	0	0
120.1	1	1	1	0	0	0	0	0
123.97	1	1	0	0	0	0	0	0
126.51	1	0	0	0	0	0	0	0
129.75	0	0	0	0	0	0	0	1
130	0	0	0	0	0	0	1	1
130.29	0	0	0	0	0	1	1	0
130.56	0	0	0	0	1	1	0	0
131.8	0	0	0	1	1	0	0	0
131.28	0	0	1	1	0	0	0	1
131.59	0	1	1	0	0	0	1	1
132.75	1	1	0	0	0	1	1	0
133.5	1	0	0	0	1	1	0	0
134.2	0	0	0	1	1	0	0	1
134.76	0	0	1	1	0	0	1	0
135.62	0	1	1	0	0	1	0	1
136.23	1	1	0	0	1	0	1	0
137.76	1	0	0	1	0	1	0	0
138.89	0	0	1	0	1	0	0	1
140.61	0	1	0	1	0	0	1	1
141.88	1	0	1	0	0	1	1	1
143.37	0	1	0	0	1	1	1	1
144.71	1	0	0	1	1	1	1	1

Idea for Sliding Window Coding Scheme

It occurred to me that setting up the neural network inputs as the current and 4 previous points of the covariance element could be improved. 

1. I read somewhere that the first element of the covariance matrix corresponds to the x position - could that be why the Minor Kidnapping datasets don't have a spike?

2. More important idea:  Dumbing the inputs down:
-Split past 10 timesteps into 5 windows (numbers are negotiable), and average the contents of each window into a single number
- Code each window into a 0 or a 1. 0 means "same or decreasing from previous window" and 1 means "increasing from last time".

Hopefully, a section where the robot is localized properly would look like this:

0 0 0 0 0

And a section where a temporary blip occurs would look like this:

0 1 1 0 0

And a section where things have really gone South would look like this:

1 1 1 1 1


And to simplify the outputs, make a similar coding scheme - threshold the normal error volume and call it a 0.  Anything over that is a 1.

Feed both good and bad (kidnapped) data coded as such into the network, and see what happens.

Neural Network Performance With Covariance Inputs

Goal: Build a NN for the Covariance Datasets

I next formatted the no-kidnapping teleop (robot driven by keyboard) data to isolate the first element of the amcl_pose covariance matrix.   (See previous blogs for description of covariance spike) First-element covariance was collected for the current and 4 previous timesteps:

Inputs (5):
Covariance[0] at Time=t
Covariance[0] at Time=(t-1) - Covariance[0] at Time=t
Covariance[0] at Time=(t-2) - Covariance[0] at Time=t
Covariance[0] at Time=(t-3) - Covariance[0] at Time=t
Covariance[0] at Time=(t-4) - Covariance[0] at Time=t

^The extra " - Covariance[0] at Time=t" is to normalize the covariance measures...

Output (1):
Error Volume:  (AMCL_X - Gazebo_X) * (AMCL_Y - Gazebo_Y) * (AMCL_QX - Gazebo_QX) *(AMCL_QY - Gazebo_QY) * (AMCL_QZ - Gazebo_QZ) *(AMCL_QW - Gazebo_QW)


I used 2 data collection files to get ~ 120 samples.
https://drive.google.com/open?id=0BwpfRdaiQmbfck43WHpfbVRQQWM
https://drive.google.com/open?id=0BwpfRdaiQmbfV1RqZldSOFN6ZXc

Sadly, the resulting model had poor performance:

Codebase:
https://drive.google.com/open?id=0BwpfRdaiQmbfbmhJX0FiVHBUM0E


To Do:
- Include the data from the other 3 data collections in the training data
- See if using the datasets collected from the programmatic driving routes do better.
- If they don't, look at the Error Volume output - see if there's a better way to do it.