4.1 point release branch (#1349)

* Prepared changelog for cherrypicked merges * Mark/1312 Consider the angle unit string from IRAM FITS images (#1313) * Consider the angle unit string for IRAM fits images * Modify the changelog * Fix scripting interface and symlink issues (#1317) * Fix scripting interface bad directory issues * Update CHANGELOG.md --------- Co-authored-by: Pam Harris <[email protected]> Co-authored-by: Adrianna Pińska <[email protected]> * Yuhsuan/1318 fix fitting mad (#1319) * calculate mad without nans * rename function; adjust doc * adjust test * update changelog * reserve array size * add log * Fix the hanging problem of region spectral profiles when a region is deleted (#1330) * Remove a mutex lock from the RegionSet method * Restore the region mutex in RegionSet and apply it around the regions map * skip running casa_data_autoupdate if checking version (#1308) * skip running casa_data_autoupdate if checking version * also skip for --help and -h and use shorter syntax * add whitespace to prevent false positives * Bumped version in app * Include casacore log messages in carta log (#1315) * Add CartaLogSink to log casacore messages in carta format * Use casacore NullLogSink for no logging --------- Co-authored-by: Pam Harris <[email protected]> Co-authored-by: Adrianna Pińska <[email protected]> * Squashed commit of the following: commit 347a6f5 Author: Pam Harris <[email protected]> Date: Wed Jan 10 07:51:59 2024 -0700 Fix style for variable names commit 13cef9c Merge: 90f305e 0407938 Author: Pam Harris <[email protected]> Date: Wed Jan 10 07:44:10 2024 -0700 Merge branch 'pam/update_carta-casacore' of github.com:CARTAvis/carta-backend into pam/update_carta-casacore commit 90f305e Author: Pam Harris <[email protected]> Date: Wed Jan 10 07:36:38 2024 -0700 Fix style of variable names commit db26557 Merge: e0cbb18 fe83785 Author: Pam Harris <[email protected]> Date: Wed Jan 10 07:30:21 2024 -0700 Merge branch 'dev' into pam/update_carta-casacore commit 0407938 Author: Pam Harris <[email protected]> Date: Fri Oct 20 16:27:30 2023 -0600 Use shared_ptr for carta log sink in casacore LogSink commit 1f65902 Author: Pam Harris <[email protected]> Date: Fri Oct 20 16:19:21 2023 -0600 Remove tabs from CMakeLists.txt commit e0cbb18 Author: Pam Harris <[email protected]> Date: Fri Oct 13 10:45:52 2023 -0600 Add entry in changelog commit 74436f7 Merge: 46a3829 e21bc8d Author: Pam Harris <[email protected]> Date: Fri Oct 13 10:12:23 2023 -0600 Merge branch 'dev' into pam/update_carta-casacore commit 46a3829 Author: Pam Harris <[email protected]> Date: Fri Sep 1 17:31:34 2023 -0600 Fix passing std::vector for casacore::Vector No longer supported in casacore update. * Updated changelog title --------- Co-authored-by: Cheng-Chin Chiang <[email protected]> Co-authored-by: pford <[email protected]> Co-authored-by: Pam Harris <[email protected]> Co-authored-by: yuhsuan <[email protected]> Co-authored-by: Anthony Moraghan <[email protected]>
CARTAvis · Jan 18, 2024 · 3b7f1e9 · 3b7f1e9
1 parent 317b39e
commit 3b7f1e9
Show file tree

Hide file tree

Showing 23 changed files with 316 additions and 159 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [4.1.0]
+
+### Fixed
+* Include casacore log messages in carta log ([#1169](https://github.com/CARTAvis/carta-backend/issues/1169)).
+* Fixed the problem of opening old IRAM fits images ([#1312](https://github.com/CARTAvis/carta-backend/issues/1312)).
+* Fixed scripting interface and symlink directory issues ([#1283](https://github.com/CARTAvis/carta-frontend/issues/1283), [#1284](https://github.com/CARTAvis/carta-frontend/issues/1284), [#1314](https://github.com/CARTAvis/carta-frontend/issues/1314)).
+* Fixed incorrect std calculation when fitting images with nan values ([#1318](https://github.com/CARTAvis/carta-backend/issues/1318)).
+* Fixed the hanging problem when deleting a region during the spectral profile process ([#1328](https://github.com/CARTAvis/carta-backend/issues/1328)).
+
+### Changed
+* Updated for compatibility with latest carta-casacore using CASA 6.6.0.
+
 ## [4.0.0]
 
 ### Changed

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -192,7 +192,7 @@ set(LINK_LIBS
 
 set(SOURCE_FILES
         ${SOURCE_FILES}
-	third-party/pugixml/src/pugixml.cpp
+        third-party/pugixml/src/pugixml.cpp
         src/Cache/TileCache.cc
         src/Cache/TilePool.cc
         src/DataStream/Compression.cc
@@ -224,6 +224,7 @@ set(SOURCE_FILES
         src/ImageStats/Histogram.cc
         src/ImageStats/StatsCalculator.cc
         src/Logger/Logger.cc
+        src/Logger/CartaLogSink.cc
         src/Main/Main.cc
         src/Main/ProgramSettings.cc
         src/Main/WebBrowser.cc

diff --git a/scripts/carta b/scripts/carta
@@ -11,8 +11,10 @@ if [ ! -z $FIRST_IP ]; then
     export SERVER_IP=$FIRST_IP
 fi
 
-if [ -x "$(command -v casa_data_autoupdate)" ]; then
-    casa_data_autoupdate
+if [[ ! " $@ " =~ ( --version | -v | --help | -h ) ]]; then
+    if [ -x "$(command -v casa_data_autoupdate)" ]; then
+        casa_data_autoupdate
+    fi
 fi
 
 carta_backend "$@"
diff --git a/src/FileList/FileListHandler.cc b/src/FileList/FileListHandler.cc
@@ -55,6 +55,9 @@ void FileListHandler::GetRelativePath(std::string& folder) {
             folder = folder.substr(1); // remove leading '/'
         }
     }
+    if (folder.empty()) {
+        folder = ".";
+    }
 }
 
 void FileListHandler::GetFileList(CARTA::FileListResponse& file_list_response, const std::string& folder, ResultMsg& result_msg,
@@ -72,65 +75,51 @@ void FileListHandler::GetFileList(CARTA::FileListResponse& file_list_response, c
         requested_folder = folder_string;
     }
 
-    std::string absolute_path(requested_folder), directory;
-
-    if (requested_folder == _top_level_folder) {
-        // Set directory relative to top (current directory). Parent is empty string.
-        file_list_response.set_directory(".");
-    } else {
-        // Normalize folder relative to top, restore path
-        GetRelativePath(requested_folder);
-        casacore::Path path(_top_level_folder);
-        path.append(requested_folder);
-
-        // Resolve path (., .., ~, symlinks)
-        try {
-            absolute_path = path.resolvedName();
-        } catch (casacore::AipsError& err) {
-            try {
-                absolute_path = path.absoluteName();
-            } catch (casacore::AipsError& err) {
-                file_list_response.set_success(false);
-                file_list_response.set_message("Cannot resolve directory path for file list.");
-                return;
-            }
-        }
+    // Normalize requested folder relative to top (top + requested = full path)
+    GetRelativePath(requested_folder);
 
-        // Set parent relative to top
-        std::string parent(path.dirName());
-        GetRelativePath(parent);
-        file_list_response.set_parent(parent);
+    // Resolve path (., .., ~, symlinks)
+    std::string message;
+    auto resolved_path = GetResolvedFilename(_top_level_folder, requested_folder, "", message);
 
-        // Set directory relative to top
-        directory = absolute_path;
-        GetRelativePath(directory);
-        file_list_response.set_directory(directory);
+    // Check resolved path
+    if (resolved_path.empty()) {
+        file_list_response.set_success(false);
+        file_list_response.set_message("File list failed: " + message);
+        return;
+    } else if ((_top_level_folder.find(resolved_path) == 0) && (resolved_path.length() < _top_level_folder.length())) {
+        // path is above top folder!
+        file_list_response.set_success(false);
+        file_list_response.set_message("Forbidden path.");
+        return;
     }
 
-    if ((_top_level_folder.find(absolute_path) == 0) && (absolute_path.length() < _top_level_folder.length())) {
-        // absolute path is above top folder
+    casacore::File folder_path(resolved_path);
+    if (!folder_path.isDirectory()) {
         file_list_response.set_success(false);
-        file_list_response.set_message("Forbidden path.");
+        file_list_response.set_message("File list failed: requested path " + folder + " is not a directory.");
         return;
     }
 
-    casacore::File folder_path(absolute_path);
-    std::string message;
+    // Set response parent and directory
+    if (requested_folder == ".") {
+        // is top folder; no directory
+        file_list_response.set_parent(requested_folder);
+    } else {
+        // Make full path to separate directory and base names
+        casacore::Path full_path(_top_level_folder);
+        full_path.append(requested_folder);
+        // parent
+        std::string parent(full_path.dirName());
+        GetRelativePath(parent);
+        file_list_response.set_parent(parent);
+        // directory
+        std::string directory(full_path.baseName());
+        file_list_response.set_directory(requested_folder);
+    }
 
+    // Iterate through directory to generate file list
     try {
-        if (!folder_path.exists()) {
-            file_list_response.set_success(false);
-            file_list_response.set_message("Requested directory " + directory + " does not exist.");
-            return;
-        }
-
-        if (!folder_path.isDirectory()) {
-            file_list_response.set_success(false);
-            file_list_response.set_message("Requested path " + directory + " is not a directory.");
-            return;
-        }
-
-        // Iterate through directory to generate file list
         casacore::Directory start_dir(folder_path);
         casacore::DirectoryIterator dir_iter(start_dir);
 
@@ -326,37 +315,38 @@ bool FileListHandler::FillRegionFileInfo(
 void FileListHandler::OnRegionFileInfoRequest(
     const CARTA::RegionFileInfoRequest& request, CARTA::RegionFileInfoResponse& response, ResultMsg& result_msg) {
     // Fill response message with file info and contents
-    casacore::Path top_path(_top_level_folder);
-    top_path.append(request.directory());
-    auto filename = request.file();
-    top_path.append(filename);
-    casacore::File cc_file(top_path);
-    std::string message, contents;
-    bool success(false);
+    auto directory = request.directory();
+    auto file = request.file();
+    std::string message;
 
-    if (!cc_file.exists()) {
-        message = "File " + filename + " does not exist.";
-        response.add_contents(contents);
-    } else if (!cc_file.isRegular(true)) {
-        message = "File " + filename + " is not a region file.";
-        response.add_contents(contents);
-    } else if (!cc_file.isReadable()) {
-        message = "File " + filename + " is not readable.";
-        response.add_contents(contents);
-    } else {
-        casacore::String full_name(cc_file.path().resolvedName());
-        auto& file_info = *response.mutable_file_info();
-        FillRegionFileInfo(file_info, full_name);
-        std::vector<std::string> file_contents;
-        if (file_info.type() == CARTA::FileType::UNKNOWN) {
-            message = "File " + filename + " is not a region file.";
-            response.add_contents(contents);
+    casacore::String full_name = GetResolvedFilename(_top_level_folder, directory, file, message);
+
+    bool success(false), add_contents(true);
+    if (!full_name.empty()) {
+        casacore::File cc_file(full_name);
+        if (!cc_file.isRegular(true)) {
+            message = "File " + file + " is not a region file.";
         } else {
-            GetRegionFileContents(full_name, file_contents);
-            success = true;
-            *response.mutable_contents() = {file_contents.begin(), file_contents.end()};
+            auto& file_info = *response.mutable_file_info();
+            FillRegionFileInfo(file_info, full_name);
+
+            if (file_info.type() == CARTA::FileType::UNKNOWN) {
+                message = "File " + file + " is not a region file.";
+            } else {
+                std::vector<std::string> file_contents;
+                GetRegionFileContents(full_name, file_contents);
+                success = true;
+                *response.mutable_contents() = {file_contents.begin(), file_contents.end()};
+                add_contents = false;
+            }
         }
     }
+
+    if (add_contents) {
+        std::string contents;
+        response.add_contents(contents);
+    }
+
     response.set_success(success);
     response.set_message(message);
 }

diff --git a/src/ImageData/CartaFitsImage.cc b/src/ImageData/CartaFitsImage.cc
@@ -475,7 +475,8 @@ void CartaFitsImage::SetFitsHeaderStrings(int nheaders, const std::string& heade
     }
 
     // For setting up image
-    _image_header_strings = no_history_strings;
+    casacore::Vector<casacore::String> const no_history_strings_v(no_history_strings);
+    _image_header_strings = no_history_strings_v;
 }
 
 casacore::Vector<casacore::String> CartaFitsImage::FitsHeaderStrings() {

diff --git a/src/ImageData/StokesFilesConnector.cc b/src/ImageData/StokesFilesConnector.cc
@@ -201,28 +201,28 @@ bool StokesFilesConnector::OpenStokesFiles(const CARTA::ConcatStokesFiles& messa
     for (int i = 0; i < message.stokes_files_size(); ++i) {
         auto stokes_file = message.stokes_files(i);
         auto stokes_type = message.stokes_files(i).polarization_type();
-        casacore::String hdu(stokes_file.hdu());
-        casacore::String full_name(GetResolvedFilename(_top_level_folder, stokes_file.directory(), stokes_file.file()));
 
         if (_loaders.count(stokes_type)) {
             err = "Duplicate Stokes type found!";
             return false;
         }
 
+        casacore::String full_name(GetResolvedFilename(_top_level_folder, stokes_file.directory(), stokes_file.file(), err));
+
         // open an image file
-        if (!full_name.empty()) {
-            try {
-                if (hdu.empty()) { // use first when required
-                    hdu = "0";
-                }
-                _loaders[stokes_type].reset(FileLoader::GetLoader(full_name));
-                _loaders[stokes_type]->OpenFile(hdu);
-            } catch (casacore::AipsError& ex) {
-                err = fmt::format("Failed to open the file: {}", ex.getMesg());
-                return false;
+        if (full_name.empty()) {
+            return false;
+        }
+
+        try {
+            casacore::String hdu(stokes_file.hdu());
+            if (hdu.empty()) { // use first when required
+                hdu = "0";
             }
-        } else {
-            err = "File name is empty or does not exist!";
+            _loaders[stokes_type].reset(FileLoader::GetLoader(full_name));
+            _loaders[stokes_type]->OpenFile(hdu);
+        } catch (casacore::AipsError& ex) {
+            err = fmt::format("Failed to open the file: {}", ex.getMesg());
             return false;
         }
 

diff --git a/src/ImageFitter/ImageFitter.cc b/src/ImageFitter/ImageFitter.cc
@@ -47,7 +47,7 @@ bool ImageFitter::FitImage(size_t width, size_t height, float* image, double bea
     _create_residual_data = create_residual_image;
     _progress_callback = progress_callback;
 
-    CalculateNanNum();
+    CalculateNanNumAndStd();
     SetInitialValues(initial_values, background_offset, fixed_params);
 
     // avoid SolveSystem crashes with insufficient data points
@@ -141,13 +141,21 @@ void ImageFitter::StopFitting() {
     _fit_data.stop_fitting = true;
 }
 
-void ImageFitter::CalculateNanNum() {
+void ImageFitter::CalculateNanNumAndStd() {
+    std::vector<double> data_notnan;
+    data_notnan.reserve(_fit_data.n);
+
     _fit_data.n_notnan = _fit_data.n;
     for (size_t i = 0; i < _fit_data.n; i++) {
         if (isnan(_fit_data.data[i])) {
             _fit_data.n_notnan--;
+        } else {
+            data_notnan.push_back(_fit_data.data[i]);
         }
     }
+
+    _image_std = GetMedianAbsDeviation(_fit_data.n_notnan, data_notnan.data());
+    spdlog::debug("MAD = {}", _image_std);
 }
 
 void ImageFitter::SetInitialValues(
@@ -221,9 +229,6 @@ int ImageFitter::SolveSystem(CARTA::FittingSolverType solver) {
     gsl_vector* y = gsl_multifit_nlinear_position(work);
     gsl_matrix* covar = gsl_matrix_alloc(p, p);
 
-    gsl_multifit_nlinear_init(_fit_values, &_fdf, work); // work-around for filling in f
-    _image_std = GetMedianAbsDeviation(_fdf.n, f->data);
-
     gsl_vector* weights = gsl_vector_alloc(n);
     gsl_vector_set_all(weights, 1 / _image_std / _image_std);
     gsl_multifit_nlinear_winit(_fit_values, weights, &_fdf, work);

diff --git a/src/ImageFitter/ImageFitter.h b/src/ImageFitter/ImageFitter.h
@@ -139,9 +139,9 @@ class ImageFitter {
     GeneratorProgressCallback _progress_callback;
 
     /**
-     * @brief Calculate the number of NaN values in the image data.
+     * @brief Calculate the number of NaN values and standard deviation of the image data.
      */
-    void CalculateNanNum();
+    void CalculateNanNumAndStd();
     /**
      * @brief Set initial fitting parameters for the fitting.
      * @param initial_values Initial fitting parameters