file updates

docs/001_Notebook_use.md

@@ -61,6 +61,9 @@ In either case, you will need to provide your UCL ISD login and password for bot
 
 We recommend that you use the 'traditional' Jupyter notebooks for this course, rather than Jupyter Lab. Both are viable options, but the notebooks, in the folder `notebooks` have features that will not work properly in Jupyter Lab. In essence, if you use the notebooks in the folder `notebooks` Jupyter Lab, the exercise answers will be exposed to you as you go through the notes, whereas they should be hidden. For example:
 
+#### Exercise 1
+ This is a test exercise
+
 In notebooks, there will be a green button that you have to press to reveal the answer, but in Jupyter Lab, the answer will be directly on show. 
 
 ## Notebooks
@@ -125,6 +128,9 @@ first_string = "hello world"
 print(first_string)
 ```
 
+    hello world
+
+
 The type of cell we use is `Code` (rather than `Markdown` above).
 
 Remember that we *execute* ('run') the code in the cell, either with the `>| Run` button above, or by pressing the `SHIFT RETURN` keys at the same time. 

docs/001_Notebook_use_answers.md

@@ -1,6 +1,6 @@
 # 001 Use of Jupyter notebooks : Answers to exercises
 
-#### Exercse 1
+#### Exercise 1
  This is a test exercise
 
 

docs/002_Unix.md

@@ -116,6 +116,7 @@ cd ~/geog0111
 ls R*
 ```
 
+    README.add.md
     README.md
 
 
@@ -130,7 +131,7 @@ cd ~/geog0111
 ls -lh README.md
 ```
 
-    -rw-r--r-- 1 ucfalew ucfa 3.7K Oct  2 18:42 README.md
+    -rw-r--r-- 1 ucfalew ucfa 3.3K Oct  5 09:38 README.md
 
 
 Here, the file size if `321B` (321 Bytes), and the file is owned by the user `plewis`. The field `-rw-r--r--` provides information on file permissions. Ignoring the first `-`, it is in 3 sets of 3 bits:

docs/003_Help.md

@@ -30,13 +30,6 @@ You can get help on an object using the `help()` method. This will return a full
 help(list.append)
 ```
 
-    Help on method_descriptor:
-
-    append(self, object, /)
-        Append object to the end of the list.
-
-
-
 #### Exercise 1
 
 * In a new code cell, type `help(list)` and look through the information provided.

docs/015_Python_control.md

@@ -189,3 +189,35 @@ We should know know how to use `if` statements in Python to control program flow
 We know that conditions inside `if` statements use indentation in Python, and we know to be careful in our use of this.
 
 There are additional notes in [docs.python.org](https://docs.python.org/3/tutorial/controlflow.html#the-range-function) you can follow up to deepen your understanding of these topics. 
+
+Summary of material in this notebook:
+
+Comparison operators:
+
+|symbol| meaning|
+|:---:|:---:|
+| == | is equivalent to |
+| != | is not equivalent to |
+| > | greater than |
+|>= | greater than or equal to|
+|<  | less than|
+|<=  | less than or equal to    |
+
+
+`If ... elif ... else`:
+
+
+        if condition1:
+            # do this 1
+            doit1()
+            ...
+        elif condition2:
+            # do this 2
+            doit2()
+            ...
+        else:
+            # do this 3
+            doit3()
+            ...
+
+

docs/016_Python_for.md

@@ -325,3 +325,35 @@ We know that conditions inside `if` statements use indentation in Python, and we
 We have also seen the use of `assert` to do some checking that our code is correct.
 
 There are additional notes in [docs.python.org](https://docs.python.org/3/tutorial/controlflow.html#the-range-function) you can follow up to deepen your understanding of these topics. You can get more practice with `assert` at [w3schools](https://www.w3schools.com/python/ref_keyword_assert.asp).
+
+
+| Command | Comment | Example | Result| 
+|---|---|---|---|
+|  `for item in list:` | loop, setting `item` to each value in `list` sequentially| see Example 1|
+| `for key,value in list_of_tuples:`|loop, setting `a,b` to each value in list of tuples | See Example 2: `list({"a":7,"b":3}.items())` | `[('a', 7), ('b', 3)]`|
+| `range(start,stop,step)` | index iterator from `start` to `stop` in steps of `step`| `list(range(1,6,2))`| `[1, 3, 5]` |
+|`enumerate(list)`| provide index of `list` | `list(enumerate(['a','b','c']))` | `[(0, 'a'), (1, 'b'), (2, 'c')]`|
+| `assert condition` | test that condition is `True`, exit otherwise | `assert True` ||
+
+**Example 1:**
+
+        for item in [1,2,3]:
+            # print item in loop
+            print(item)
+
+Result:
+
+        1
+        2
+        3
+
+
+**Example 2:**
+
+        for key,value in {"a":7,"b":3}.items():
+            print(f'key is {key}, value is {value}')
+            
+Result:
+
+        key is a, value is 7
+        key is b, value is 3

docs/017_Functions.md

@@ -591,3 +591,25 @@ The use of `**kwargs` can be useful sometimes, as you can more easily keep track
 In this section, we have learned about writing a function. We have seen that they generally will have zero or more input positional arguments and zero or more keyword arguments. They will typically return some value. We have also seen how we can define a `doc string` to give the user information on how to use the function, and also how we can use `assert` to build tests for our codes. We have been through some design considerations, and seen that it is best to plan you functions by thinking about the purpose, the inputs and the outputs. Then, for the core code, you just need to develop a skeleton code and docstring structure, test that, and insert your core code. You should think about modifications using keyword arguments that you might want to include, but these will often come in a second pass of development.
 
 When we write Python codes from now on, we will often make use of functions.
+
+Remember:
+
+Anatomy of a function:
+
+        def my_function(arg1,arg2,...,kw0=...,kw1=...):
+          '''
+          Document string 
+          '''
+
+          # comments
+
+          retval =  ... 
+
+          # return
+          return retval
+
+
+Also written as:
+
+        def my_function(*args,**kwargs):
+

docs/018_Running_Python.md

@@ -91,7 +91,6 @@ Another thing we can do is to `import` the code from the Python file into Python
 
 ```python
 from geog0111.helloWorld import helloWorld
-
 helloWorld()
 ```
 
@@ -579,3 +578,55 @@ We have also seen how we can use
 in a bash script to generate and document our Python files, though we would also typically edits the file through some text editor.
 
 We have seen how to run a Python script from a notebook, using `%run` or via a bash shell with `%%bash`. We have seen that to 'run' a Python script, we need to change the file permissions iusing `chmod`, either in `bash` or using `Path().chmod()`. We have used the octal code `755` to change the file permissions for execution.
+
+Remember:
+
+
+
+|Command| Comment| Example| Result|
+|---|---|---|---|
+|`!unix_cmd` | Run unix command `unix_cmd` from Jupyter code cell | `!geog0111/helloWorld.py`| `hello world`|
+|| | `!ls -l geog0111/helloWorld.py`| `-rwxr-xr-x 1 ucfalew ucfa 514 Oct  1 13:10 geog0111/helloWorld.py`|
+|`%%bash` | Turn Jupyter Python code cell into `bash` cell | `%%bash` | | 
+| | | `chmod 755 geog0111/helloWorld.py`
+| | |`ls -l geog0111/helloWorld.py` | `-rwxr-xr-x 1 ucfalew ucfa 514 Oct  1 13:10 geog0111/helloWorld.py`|
+| `%run script.py` | Run Python script `script.py` from Jupyter code cell| `%run geog0111/helloWorld.py` | `hello world`|
+|     `cat << XX > YY; XX` | Put  marker `XX` in bash script and send text up to `XX` into file `YY` | `cat << EOF > work/file.py` |
+|||    `hello world` | `cat work/file.py`
+|||    `EOF` | `hello world`|
+
+Form of a Python script:
+
+
+            #!/usr/bin/env python
+            # -*- coding: utf-8 -*- 
+
+            '''
+            helloWorld
+
+            Purpose:
+
+              function print the string 'hello world'
+
+            '''
+
+            __author__    = "P Lewis"
+            __copyright__ = "Copyright 2020 P Lewis"
+            __license__   = "GPLv3"
+            __email__     = "[email protected]"
+
+            def helloWorld():
+                '''
+                function to print the string 'hello world'
+
+                '''
+                print('hello world')
+
+
+            # example calling the function
+            def main():
+                helloWorld()
+
+            if __name__ == "__main__":
+                # execute only if run as a script
+                main()

docs/020_Python_files.md

@@ -20,11 +20,11 @@ You will need some understanding of the following:
 * [002 Unix](002_Unix.md) with a good familiarity with the UNIX commands we have been through.
 * [003 Getting help](003_Help.md)
 * [004_Accounts](004_Accounts.md)
+* [005_Packages](005_Packages.md)
 * [010 Variables, comments and print()](010_Python_Introduction.md)
 * [011 Data types](011_Python_data_types.md) 
 * [012 String formatting](012_Python_strings.md)
 * [013_Python_string_methods](013_Python_string_methods.md)
-* [018_Packages](018_Packages.md)
 
 ### Test
 

docs/021_Streams.md

@@ -1869,3 +1869,27 @@ In this section, we have used `Path` and `URL` classes to open streams from file
 We have seen that using these object-oriented classes to deal with files and URLs means that we can use essentially the same functions throughout. 
 
 We have come across the `pandas` package for reading tabular and similar datasets.
+
+
+`pandas`:
+
+| Command | Comment | 
+| --:|---|
+|`pd.read_csv(f)`| Read CSV data from file or URL `f`|
+|`pd.read_table(f)`| Read table data from file or URL `f`|
+| `skiprows=N` | Keyword to skip `N` rows in reading for `pd.read_table`|
+| `na_values=[-99.9]` | Keyword to set list of values to ignore (`-99.9` here) |
+| `sep` | Keyword to define field separator |
+| `engine='python'` or `engine='C'` | Keyword to set reading engine. `python` is more flexible, but `C` is faster. |
+|`df.transpose()` | Transpose (rows->columns, columns->rows) pandas dataframe `df`|
+|`df.head(N)` | first `N` lines of data (default 5) |
+|`df.columns` | list-like object of column headings |
+|`df[cname]` | Select column with name `cname`|
+|`df[[c1,c2,c3]]` | Select columns with names `c1`, `c2` and `c3`|
+| `pd.DataFrame(list,columns=cnames)` | Create `pandas` dataframe from information in list-like structures `list` with names from list `cnames`|
+|`pd.to_datetime(str_list)` | convert list of date strings (e.g. of form `YYYY-MM-DD`) to `datetime` representation |
+| `df[datetimes].dt.month` | month from `datetime` field fromn `datetime`-format column with name `datetimes`|
+| `df[datetimes].dt.year` | year from `datetime` field fromn `datetime`-format column with name `datetimes`|
+| `df[datetimes].dt.day` | day from `datetime` field fromn `datetime`-format column with name `datetimes`|
+|`df.to_csv(filename,index=False)` |Write dataframe `df` to CSV format file, with no index column|
+

docs/022_Read_write_files.md

@@ -279,3 +279,32 @@ In this section, we have used `Path` and `URL` classes to read and write text an
 We have learned how to use `gdal` to look at the sub-datasets in an HDF file and also how to read them.
 
 You should now have some confidence in these matters, so that if you were set a task of downloading and saving datasets, as well as other tasks such as finding their size, whether the exists or not, you could do this. 
+
+Remember:
+
+Modis library
+
+            from  geog0111.modis import Modis
+            modis = Modis(**kwargs)
+            
+
+            get_url(**kwargs) method of geog0111.modis.Modis instance
+                Get URL object list for NASA MODIS products
+                for the specified product, tile, year, month, day
+
+                Keyword Arguments:
+
+                verbose:  bool
+                product : str e.g. 'MCD15A3H'
+                tile    : str e.g. 'h08v06'
+                year    : str valid 2000-present
+                month   : str 01-12
+                day     : str 01-(28,29,30,31)
+
+`gdal`
+
+| Command | Comment |
+|---|---|
+|`g = gdal.Open(filename)` | Open geospatial file `filename` and return `gdal` object `g` (`None` if file not opened correctly)|
+|`g.GetSubDatasets()` | Get list of sub-datasets from `gdal` object `g`| 
+|`g.ReadAsArray()` | Read dataset from `gdal` object `g` into array |

docs/023_Plotting.md

@@ -928,3 +928,56 @@ If you want to go further towards re-creating this, you consult [the matplotlib
 In this section, we have learned how to plot graphs from datasets we have read in or downloaded from the web. We have concentrated in simple line graphs, with possibly multiple sub-plots here, although there are many other types of graph you may consider. 
 
 The [`matplotlib` gallery](https://matplotlib.org/3.3.1/gallery/index.html) provides a useful starting point for other types of plot. You might also consider the Python packages [`bokeh`](https://bokeh.org/) and [`seaborn`](https://seaborn.pydata.org/) for improved visualisation and interaction.
+
+Remember:
+
+
+
+            import matplotlib.pyplot as plt
+            import matplotlib.dates as mdates
+
+
+| Function | Comment | Keywords |
+|---|---|---|
+|`fig, axs = plt.subplots(xshape,yshape)` | set up new figure as array of sub-plots `xshape` by `yshape` and return `fig` object for figure and `axs` object for subplots. `axs` is array of objects shape `(xshape,yshape)`, or single object if `(1,1)`. We often use `axs = axs.flatten()` to flatten the `axs` array to 1D.|`figsize=(sx,sy)` : set figure size `sx` by `sy`|
+| `plt.savefig("ofile.png")` | Save plot to file `"ofile.png"` | 
+| `fig.suptitle(name)` | set title `name` at top of figure `fig`
+| `im = axs.plot(xdata,ydata,cs)` | plot line graph with `ydata` as a function of `xdata` and return plot object `im`. If argument `cs` is given, this is a colour/symbol e.g. `k+` for black crosses. | `label=str` : set `str` as label for this line. For use with `legend`| 
+| `im = plt.errorbar(xdata,ydata)` | Error bar plot for `ydata` as a function of `xdata`. Needs kwarg | `yerr=yerr` : set y-error bar to values in array `yerr` |
+| `axs.set_xlim(x0,x1)` | Set plot x-extent to from `x0` to `x1` |
+| `axs.set_ylim(y0,y1)` | Set plot x-extent to from `y0` to `y1` |
+| `axs.set_title(name)` | set sub-plot `axs` title to `name` |
+| `axs.set_xlabel(xlab)` | set x-axis label to `xlab` |
+| `axs.set_ylabel(ylab)` | set y-axis label to `ylab` |1
+| `axs.legend()` | set legend in plot | `loc='best'` : best location, otherwise `top left` etc. |
+| `mdates.MonthLocator()` | `MonthLocator` object of month locators, e.g. `months = mdates.MonthLocator()` |
+| `axs.xaxis.set_major_locator(months)` | set major ticks on x-axis to `Locator` object of `months`|
+
+
+Some colours and symbols:
+
+| symbol/colour | name |
+|---|---|
+|'k+' | black plus |
+|'r.' | red dot  |
+|'go' | green circle  |
+|'b-' | blue line  |
+|'c--' | cyan dashed line  |
+|'y-o' | yellow  line with circles  |
+
+
+`datetime`
+
+            from datetime import datetime
+            
+            
+| function | comment |
+|---|---|
+|`datetime.now()` | date and time for now |
+|`datetime(year, month, day, hour, minute, second, millisecond)` | return `datatime` object for time/date specified. Not all fields need be given. Can also use keywords but `year`, `month` and `day` must be given e.g.: `datetime(2020,day=1,month=2,hour=12)`|
+| `datetime.day` | day etc. |
+| `timedelta` | subtract two `datetime`s to get a `timedelta`|
+|`timedelta.days` | number of days in `timedelta` |
+| `dt.strftime("%H:%M:%S")` | represent `datetime` object `dt` as `hour:minute:second`|
+| `dt.strftime("%m/%d/%Y")` | represent `datetime` object `dt` as `month/day/year`|
+| `datetime.strptime(str,format)` | load string `str` into `datetime` object interpreting as format `format`, e.g. `datetime.strptime("11 November, 1918", "%d %B, %Y")` -> `1918-11-11 00:00:00`; e.g. `datetime.strptime("2020-06-20", "%Y-%m-%d")` -> `2020-06-20 00:00:00`|